Evolving process: IBC Accelerator on using a live Formula E test to push the use of edge compute and containerised applications forward

Ian Wagdin, BBC R&D senior technology transfer manager, at Formula E for the IBC Accelerator Connect and Produce Anywhere live test

At the recent Formula E London E-Prix at Excel on 20 July, IBC Accelerator Connect and Produce Anywhere (CAPA) was on site showcasing how a full live broadcast can be encapsulated entirely on software hosted on an edge compute cluster.

As well as project champions BBC R&D and Channel 4 on site, tech partners TSL, Lama, Vizrt, Zixi and Techex were on the ground making sure everything worked smoothly.

CAPA is evolving and in its second year as an IBC Accelerator. Ian Wagdin, BBC R&D senior technology transformation manager, spoke to SVG Europe amid the whine of Formula E cars warming up on the track: “Last year we set a vision, did a little bit of testing, and this year we’re very much trying to verify and validate against that vision, which means putting the kit out there. Now in part two of CAPA, we’re trying to do a few more kind of field trials and tests in real world environments. Very kindly Formula E invited us to come down alongside Google to run our edge compute mode at a large sporting event.”

Wagdin outlines CAPA’s first year: “We set quite an ambitious vision last year on what we wanted to see around using edge compute, trying to understand the benefits, trying to understand the ways that we might deploy it, why we might deploy it, and also parallel pieces of work going on in the EBU and various other bits looking at software-based production systems. So it’s part of a wider trend to move away from big hardware specialist broadcast stacks, to more underlying IT-based commodity technologies where the media operations then go over the top of that standardised IT stack.

“What we very quickly realised last year when we started to do some testing of real world deployments was that actually it’s all very well to talk about it, but when you try and do it for real, there’s a whole bunch of things that we need to get into place and things that we don’t understand from a commercial perspective, or things that take slightly longer because the IT world works in a slightly different way to the way we work in the broadcast world. Also, manufacturers aren’t yet geared up to deliver their software in a way that is deliverable in containers,” he notes.

Containerised apps

Defining containerisation, Wagdin explains: “Literally you box up an application, and it doesn’t need to be your entire product, it can be components of your product. We’ve seen big monolithic single vendor solutions in the IP world, which has moved away from what we require in the broadcast domain of best of breed, multi-vendor solutions where we can have the best vision mixer depending on operational skills and the best replay engine, the best graphics engine, or whatever. Trying to do all of that within one technology stack quite often means that you are locked into one vendor or you’ve got to go by some convoluted route to move from one vendor to another.

“Containerisation means that I can effectively get to the point where if I can just get the containers to talk to each other or I can basically have a standard language that enables me to deploy these things, I can deploy them really quickly. In an ideal world, I’ll be able to licence them really quickly so I can deploy a compute cluster out on location and say I want five channels and then just licence the sixth channel or seventh channel as a camera comes up. And then in the same way as we want to be able to do in the cloud, just spin it down when we’re not using it to save resources.

“So that’s a massive move away from what we do now, which is build a 50 channel OB truck and park that somewhere with all the resources going into building that,” continues Wagdin. “Then a lot of the time the truck is in doing maintenance or it’s being reconfigured and stuff like that. Whereas with [containerised applications] we can reconfigure it easily because it’s software.”

Not quite there yet

Today manufacturers are not quite there yet on being able to offer products that work within the future-focused edge computing environment of distributed sports broadcast production that CAPA – and Wagdin – are dreaming of.

Wagdin says: “Manufacturers have developed software versions of their product, but they tend to run those on virtual machines, which in a cloud or an edge environment isn’t necessarily the optimal way that you might want to deploy those solutions, especially in a multi-vendor scenario. Ideally you want to containerise all your different applications and then run them on your IT stack and then get them all to talk to each other. That’s really easy to say, a whole world of pain to do. So that’s really what we got to last year, where we had virtual machines running in this configuration.

“This year we’re starting to move more towards containerisation,” he continues. “So we’ve got – I think on our cluster – over a hundred containers running. Most of those are doing the software or the IT management stuff and then we’ve got one containerised media application, and then we’re pushing to move other people away from virtual machines and into containers.”

Evolving process

Wagdin says this is all part of an evolving technological process, which began with IP and the 2110 standards and work on NDI – which was aimed at moving broadcasting away from SDI hardware-based infrastructure – into an IP-based world. However, that still involves lots of kit and trucks.

He notes: “Part of that vision for IP was always to enable us to do these more software-centric standardised compute-type workflows. We’ve seen a lot of companies – including Gravity who are hosting us here at Formula E – who are enabling us to do more remote production. Where we’re sitting now, just four years ago this would’ve been full of broadcast trucks with local broadcasters or rights holding broadcasters, all creating their own content.

Formula E’s broadcast production is broken down into portable pods that are shipped around the world for its races. Here, they sit in the media compound at the London E-Prix at Excel, London on 20 July 2024

“Now all that’s here is a few control rooms and [tech pods] that they fly around the world, and everything then goes back to Gravity at Westworks. So it doesn’t matter where they’re producing from – whether it’s Berlin, London, wherever they are – it all works in the same way.

What IP enables them to do is to basically make different feeds available for all the host broadcast as well as rights holding broadcasters here on site. They’ve got a far more flexible sustainable model now and they’re doing remote production and that’s great. But in order to deliver that, they’ve got some pretty hefty connectivity here that enables them to push all of their cameras out to W12, and they’ve got some latency built into that system – quite surprisingly low amounts actually.

“This is what characterises a lot of remote production, a lot of cloud production, is that you need that connectivity and the larger your event gets, the more connectivity you need,” he says.

However, CAPA is focused on helping to produce events where connectivity is poor, states Wagdin. “Where we set out with the Connect and Produce Anywhere was not looking at events like this or Premier League or whatever where you’ve got really good connectivity, but actually there’s a whole bunch of other things that we do where we don’t have good connectivity and you don’t want to spend lots of money on connectivity because you’re only there for a day a week. While [Gravity and Formula E’s] model really works for large scale events with big worldwide audiences, it doesn’t work for a basketball match, it doesn’t work for a Gaelic football tournament, it doesn’t work for all of these things that we want to cover more of and we’ve got platforms to distribute them on.

“Now we’ve got IP-based distribution and we’ve got this demand for more and more live content, but it’s actually really hard to produce that without still sending even a small truck with all of the manpower. So the idea behind the CAPA project is that we can literally just send out the compute power and then we can control that from anywhere,” he states.

Access all areas

Back to the weekend of testing, Wagdin says: “Now here, because it’s easy [for connectivity at Excel], we’re choosing to operate it from right next door to the Google Distributed Cloud edge compute cluster, but that’s just an ethernet connection that could be done over whatever internet you’ve got available, whether that’s LEO, whether that’s 5G, whether that’s other sorts of broadcast connectivity or non-broadcast connectivity. So we can minimise down the amount of bandwidth we require, but still maintain the same quality because actually all of the operational stuff is happening on the compute side, but the control side is separated from that, which then means we can operate from the location remotely or from our broadcast centres, and we can choose to where that is depending on the editorial demand or the topic of event.

“It also means that, because all those sources are available on that cluster, we can enable different teams to have different access to all of the same content. So if you’ve got a web team who are producing content for the web, they can get access to all of your cameras rather than just whatever you choose to give them.”

CAPA will be presenting some of its learnings at the IBC Show in Amsterdam in September