Level change: BT Sport reaches for the cloud in next phase of its decentralised remote production evolution

BT Sport is producing its Fight Week Live: UFC London preview show in the cloud

BT Sport is embarking on the second stage of its decentralised remote production evolution with a radical move to the cloud.

The broadcaster is expanding its use of cloud production as it further enhances its use of innovative ways of working to reduce its carbon footprint, increase inclusion, and continue pushing for more flexible and sustainable ways of creating live sports TV.

On the overarching strategy with cloud technology for BT Sport, Jamie Hindhaugh chief operating officer BT Sport, tells SVG Europe: “This is an extension of our ever-going journey around remote production, and specifically around two elements, our opportunity to continually push and drive the sustainability agenda, and also more and more look at inclusion, ie, people can work in different locations, therefore you can be more inclusive of the team you pull together.

“But [we’re] also looking at further opportunities [with cloud], so at the moment on remote production, we’ve got around about 10 different ways of doing remote depending on the venue, depending on the rights we’re covering, etc. It’s been very successful, [and we’re] very, very pleased with that, and we’re seeing – on the whole – a sustainability impact of about 50% reduction on some key events.

“But [we’re also looking at what other] opportunities there are around cloud – first understanding what cloud is – and also understanding the capabilities of cloud [and therefore what will be] the sensible next step to further push on both those sustainability and inclusion agendas,” he says.

Taking UFC into the cloud

Tonight BT Sport is producing its Fight Week Live: UFC London preview show in the cloud. The live preview show with UFC president Dana White and BT Sport’s Michael Bisping, Caroline Pearce, Adam Catterall and Nick Peet, is running ahead of the UFC’s return to the O2 Arena this Saturday.

“We’ve been doing this for a little while and we’ve been starting to look more and more at the application of cloud tools as the next generation, next step on from our COVID-induced remote production transformation”

Coming from the top of the BT Tower with 50 guests, the preview show will include a live audience and a five-camera set up, using Mavis cloud technology to get the production on air to BT Sport 2 HD and YouTube.

BT Sport chief engineer Andy Beale, says: “The director wants to be close, so the director’s actually going to sit one floor above on the tower, with his iPad and his fallback and just carry on. He could literally be anywhere, but he wants to be close to help us out with the shots, which works really well.

“The producer, the PA, and the graphics will be in Stratford, but again, they could be at home; it’s just a really nice, simple, robust model. It’ll be kept on the public internet from the tower into the cloud and onto both social, onto YouTube, and onto our linear TV platform, BT Sport 2 HD, which is really exciting.”

Experimental phase

BT Sport has been working on its cloud agenda in an experimental phase for almost half a year. Hindhaugh states: “We’ve been playing in the cloud world now, for want of a better term, for about six months. As you’ll have seen previously, programmes like the Ashes Daily show were fully produced in the cloud.

“We’re using that [kind of test] twofold: firstly, it’s to understand the workflows, with Andy [Beale CTO at BT Sport] sense checking what does and doesn’t work. But secondly, it’s allowing us to fine tune and explore the opportunities that cloud brings. So Andy’s been doing some brilliant work with the Premier League, for instance, on an IBC Accelerator Challenge: Sustainability in Production.”

Beale says: “We’ve been doing this for a little while and we’ve been starting to look more and more at the application of cloud tools as the next generation, next step on from our COVID-induced remote production transformation. And, as we mentioned at [SVG Europe’s Football Summit 2022], that came to a head in a bigger way around our IBC accelerator demo, working in collaboration with the BBC, Sky, and Premier League, where we took off chunks each and tried to break down the bigger jigsaw puzzle of a big Premier League game, trying to explore interoperability, trying to explore security, trying to explore the production tool set, is it good enough, is it fast enough, is it high enough quality?”

From the IBC Accelerator Challenge BT Sport amassed a 77% power reduction versus a similar production using its previous remote technology system, which, says Beale, “is massive, absolutely massive”.

Beale says the power savings with cloud are exciting. “In the old days everyone left all their [kit] just turned on the whole time, so it’s not particularly sustainable, whereas this is, “since you finished, turn it off”. As much of the opportunity [with cloud SaaS] is what is not turned on, as much as what it is turned on, so that’s also a really big win.”

“The ability to genuinely – that easily – just change location and carry on putting out a first class show, shows how powerful the opportunity around the cloud is”

Talking through what BT Sport means by ‘cloud’, Beale says: “We learned lots in that trial [IBC] demo,” says Beale. “We learned lots of things about interoperability that we wanted to learn, but [to] then push ourselves on a bit further [we needed to do more trials]. We wanted to move forward and try the use cases and try different things.

“What’s really, really interesting is that, much like all the different remote workflows where we used to have a combination of full remote, what we call surface remote, hybrid remote, and satellite remote, the cloud is very similar in that we have different tools, and different models that work really well for different types of products.

Ashes Daily trials and tests

“To give you a couple of examples, we did the Ashes Daily show, which was the morning breakfast show that discusses and picks over the absolute carnage that England have suffered the night before, for several weeks over Christmas. That was the first time we tried to do a full cloud show,” notes Beale.

“For that, we actually chose a cloud product, which is software is a service (SaaS) system called Mavis, who are an interesting little company, working with Singular Live to provide live graphics, and we ran that pretty successfully for all five tests. All the cameras and microphones were on location in Fitzrovia [London] at the Sixes Cricket Club that we were filming in, connected to the public internet locally on their public internet service, and all into the cloud.

“All of our production team started off working together our production pods, where it’s one person per pod, and you can [configure each pod to] do anything; they can be EVS, they can be graphics, they can be an AP producer,” Beale goes on. “We converted four of those pods into various different functions for the cloud, so the director sat in one on his iPad, a producer sat in another, the graphics operator sat in another, and the PA taking all the counts and liasing with playout sat in another, and that worked really, really well. All of the actual production is happening in the cloud, and it just so happens, they physically happened to be sat side by side.”

However, when the Omicron COVID variant came along at Christmas – just as the team were working on the cloud test on the Ashes Daily – most of the crew tested positive for the virus, although they had no symptoms. The team left their pods at BT Sport and went home, but took their small surface kits with them to various locations across the UK and carried on working on the show over the cloud.

Beale goes on: “So Boxing Day, which is arguably our biggest show, because everyone’s at home watching, the director was in Surrey, I think the graphics AP was in Cardiff, and everyone just literally picked up their little bit [of kit] and took it home with them, and just carried on. And what’s amazing is the viewer at home would never know.”

Beale says the cloud technologies have the ability to revolutionise remote production. He explains: “The real enabling difference between our traditional hardware production tools, which are still based around old legacy broadcast hardware, and are very expensive, pretty bulky, and you can’t really take it [home] easily – I mean, we did in the first phase of the pandemic when we sent all that amazing kit home, but you can’t really do that in a sustainable way – [and cloud is] this is literally a laptop and an iPad and a pair of headphones.

“The ability to genuinely – that easily – just change location and carry on putting out a first-class show, shows how powerful the opportunity around the cloud is,” Beale continues. “And we just carry on. The last two tests of cloud production on the Ashes Daily] were arranged in that way. No one came into Stratford again because they didn’t need to, but [the show] still looked great and it was still on air, so that was really, really exciting.”

The Ashes Daily Show test was created using a SaaS model, which would enable the broadcaster to work out a price per event cost using the tools as and when they were needed. Notes Beale: “The brilliant thing about cloud is you can just turn it on and off when you need it.”

Revolutionise remote production

However, BT Sport also intends to trial a more complex cloud set up; The Ashes Daily Show involved a relatively simple set up, Beale notes, with four cameras, four microphones and a little bit of VT. States Beale: “We then started looking at how we could do more complicated production and live match coverage production.

“So we’ve been working on [Gallagher Premiership Rugby matches output for the League’s website], and we are running an integration ourselves on the cloud. So we’re running in an AWS instance that we are in control of, deploying tools like Inspector, Simply Live, VMix, again Singular for graphics. We’ve done eight camera rugby coverage pretty successfully, totally remote again, and again, those control services now are so much more accessible and affordable, but they still have the same tool set.”

Beale gives an example of how flexible and intuitive the cloud technologies are for crew to manipulate: “When we did the audio arrangement for this [Gallagher Premiership Rugby] test, we were using just Behringer control surfaces, which costs £450, plus the traditional sound surface, which is costing tens of thousands of pounds. Because the clever bit, the software in the back end, is now just a control panel but in the old days it would’ve been a hardware sound mix. So the sound supervisor came in for the rehearsal day and he started tinkling and he was like, “I’ve got one of these at home, actually”. He asked how it was set up, [then left]. We were still there in evening doing a few final set ups – things you do, engineers, tinkering away – and suddenly all this music came blasting out the speakers, and this guy had got IT to add him to the wireless team and get him onto the VPN and everything, and he just started doing it from home!”

Beale continues: “My point is, this was a much more complicated production – this was a 16 fader production, audio stems – this wasn’t just four mics and a bit of bubble and one BT source, this was a much more complicated thing, with pitch mics and mixing, but it had the same agility and it had the same real flexibility [as a simple production].

“That demonstrates, quite nicely, the opportunity [with cloud], and again, turn it on or off and have as many as you want,” Beale says.

Following the Gallagher League trials, BT Sport is moving its now production for the League to the cloud. The League has been upgrading all network connectivity to the stadiums involved over recent months to create dual-redundant 10GB services, “for complete redundancy and high-speed bandwidth at the venues,” says Beale.

During that upgrade period, BT Sport has produced shows on a hybrid remote and on-prem system while continuing to test cloud services, but with the network upgrades almost complete, that production will be moving into the cloud onto the pre-tested technologies BT Sport has been working with previously, “and will probably run out of High Wycombe at the EMG ROC,” says Beale.

Another cloud test carried out by BT Sport’s Beale and his team was with the Scottish rugby team, doing a live behind the scenes look at the team’s training sessions for Scottish secondary school children. The cameras went into the team’s Six Nations training camp at Oriam, Edinburgh to see their preparations and more.

Says Beale: “We were actually able to get Scottish rugby school kids live into their Six Nations training camp, and see all of their preparation, that build up, get some interviews, see their exercises, see their recovery plan, go behind in their gym, go to their hydrotherapy pool, and all this stuff. And that was a live show.”

The production involved six cameras, lots of mics and graphics, “done in this new [cloud] model, where we just had four people – this time they happened to be sat at home but they could’ve literally been sat anywhere – making a show which went out for an hour and a half during half term. It just shows the power and the speed at which we could do this stuff; it’s really exciting,” Beale enthuses.

Changing mindsets

While cloud SaaS can save money versus buying, running and maintaining hardware, the OPEX for the cloud or CAPEX for hardware question will require broadcasters to create stronger relationships with their finance teams in order to keep everyone happy, notes Hindhaugh.

He explains: “The problem is with our industry as a standard, it’s very much around CAPEX, hardware and OPEX, operational. All of us are going to have to work with our finance support teams to get them to understand [the changes cloud use will bring to that equation]. The problem is [with cloud technology usage] OPEX becomes much more flexible, going up and down. When you’ve got CAPEX, you smooth out your costs.

“So the ironic thing is, you will save money [with cloud versus traditional broadcast systems] by moving to this OPEX software model, but from a forecasting perspective and from a business perspective, that creates uncertainty, which people are not used to.”

Concluding on what this means for BT Sport’s remote production plans, Hindhaugh says: “It’s two things. Andy’s point about turning things on and off is really, really important. Again a point Andy makes – and it’s a game of getting people to adapt – but the geographical lack of handcuffs as regards geography starts making a real difference, both from UK perspective, but also wider afield.

“We are determined to become a more inclusive broadcaster and we don’t want geography to become a limiter,” says Hindhaugh. “So [with cloud-based remote production] you start enabling less travel, more flexibility for how people work, which gives more people the opportunity [to work], while using less energy. It’s a win-win-win all the way through, from a sustainability and inclusion perspective.”

Watch Fight Week Live: UFC London live with UFC president Dana White and BT Sport’s Michael Bisping, Caroline Pearce, Adam Catterall and Nick Peet on BT Sport 2 HD and YouTube at 5pm (UK time)