Adventures in IP production: Sky Sports maximising value through innovation
Sky Sports’ Clement, Ruddell and Roxburgh at BVE2016 in London
Sky Sports has been trialling IP production, moving from limited tests to, most recently, the remote production of a three-day world championships, each time adding a little complexity and learning a lot from the experience. However, it is risky, particularly as: “we all need to spend efficiently, but current TV workflows work,” James Clement, Sky Sports’ Head of Operations, told a BVE seminar: ‘Adventures in IP production – maximising value through innovation’.
“Within sport, we’re striving to ensure every penny we spend provides the maximum impact and value to our viewers, and this means we’ve had to make choices,” he said. This has meant asking of every event and show it makes: Can we do it differently? Can it make savings that can be re-invested in programming, and improving the viewer experience?
Part of that process involves the production teams. “We will change their workflow,” although he tries to minimise that. “You can’t do remote production of the US Open tennis if the whole team is still going to be stationed in New York. They have to compromise too. But the net effect should be imperceptible to the viewer. In fact, we’ve often created the opportunity for increased production value.”
Sky has been testing some IP innovations on Formula One over the last three years. “When you are flying tonnes of cargo around the world, the motivation is clear to find better ways of making it efficient,” he added.
Mike Ruddell, Sky Sports’ Head of Technology, believes that IP is just the latest tool, but recognises that it can offer the production team extra facilities, and ultimately a better experience for viewers. One show that has benefitted from IP is Sky Sports Monday Night Football, which has virtual sets and graphics, with analysis touch screens. It was one of the key test beds for IP, including voice-over-IP for communications and self-delivered video-over-IP.
For talkback, “using VoIP codecs has allowed us to use raw IP data circuits, which are much cheaper than ISDN and achieve far higher audio quality, giving the guys a better experience,” said Ruddell. It is also using video-over-IP for ISO camera feeds, some of which are used for half-time studio edits, as well as three live HD streams that are stitched back together in the studio to give a super-wide live vista of the stadium on screen behind the presenters.
The savings from self-encoding the video and using cheap IP circuits for this, rather than using the standard telco encoding, “has very quickly paid back the CapEx we’ve had to invest in buying those video codecs ourselves,” he added.
Sky Sports Soccer Saturday covers live games that it doesn’t have the live rights to, and “lives and dies by its connectivity,” with about 30 reporters around the country. With the advent of lightweight satellite systems and bonded cellular links, Sky can now deploy IP video and audio for reporters where they’d previously been lucky to get a good audio feed. It also uses IP ISO camera feeds for shots of crowd reactions at matches, which are shown in the background in the studio.
ATP Tennis: Out of the comfort zone
Sky produces about a dozen tennis tournaments a year from its studios, and twice a year takes the team on the road, “out of their comfort zone, and out of their workflow zone too, so it’s not their gallery, not their edit, not their workflow,” said Gordon Roxburgh, Technical Manager, Sky Sports.
The challenge was to give that team some of the facilities they were used to at Sky, outside Sky. The first was at the US Open Tennis 2014, where it installed a large data pipe from the OB directly back to Sky, which gave them extra vision circuits, full VoIP communications, and the ability to move media both ways, using WAN file acceleration techniques. Roxburgh also saw this as an opportunity to undergo a proof of concept with Snell, with Alchemist OD software, to do an edit at Sky at 50i and get it over to the US in 59.94i without any additional work.
The next step, at the ATP finals in London, was to do all editing back at base. “Why build an expensive edit workflow and edit platform at the outside broadcast when you have all those facilities at Sky?” asked Roxburgh.
In 2015, they left the entire production team for the US Open in the UK. “We felt we could do a full remote production for the 15 days,” (with up to 12 hours on air each day) using five cameras live from the US and bringing back the 12 televised show courts from the host broadcaster. “This workflow kept the majority of the production team in their environment, where they produce tennis week in, week out, for Sky. It also let us better resource the outside broadcast, to put extra cameras court-side, so we could cross to our reporters,” and use its Sky Pad, the high-bright touch screen used for F1, so players could explain their matches.
Sky used NTT low-latency H.264 encoding, with eight feeds from the venue, four feeds back (including studio output for the presenters on an RF link across the venue — so they could see themselves on air less than a half second later).
“The complex bit we had to work with was the communication piece. It’s very easy to put cameras on to an IP circuit and bring it back. To make a programme, that’s where the level of complexity grows,” he added. “We ended up with ten four-wire communications circuits, and about eight IFB (earpiece feeds) to the presenters,” essentially by connecting all of its main talkback systems together. It also deployed its accelerated file transfer to send clips from the gallery to the Sky Pad.
It was a “massive success,” which was built on at the subsequent ATP event in November by allowing its OB personnel to browse Sky’s media asset management system from the venue, accessing any clips directly. It also added further connectivity for editing and for social media applications. “We had the team working out at an OB with the true workflow and resources as if they were at a gallery within Sky.”
Remote control: The paths back to base
Last May, as a test, Sky worked to produce “an OB-quality cut” of an under-21’s soccer match from Manchester back at Osterley, with feeds of each camera coming back on multiple IP streams. “The coverage we created was, in essence, the same as the one that went out from the truck on site,” said Jonathan Craggs, Senior Technical Supervisor, Sky Production Services.
During a second test it also took in the three hi-motion phases from the super-slomo camera back over three separate IP streams, “to see how a remote server, back at base, would handle the transport.” Using IP also gave them a return path, extending studio tallies to the ground.
Jonathan Craggs: “the low-latency IP paths were coming back to us a lot quicker than the unilateral world feeds”
For the US Open Tennis, in August, it was crucial to make the connection as transparent as possible. “It was never going to be feasible to have a one-to-one ratio between the local sources on site in New York and the paths back to base, so we had to devise a system where we’d have modes of operation depending on where the presenters were, or the type of analysis being done, which was all switched to us by the local end,” he explained.
Craggs believes that audio has taken the biggest leap into IP of all the crafts. For its first two football tests, “we got the 44 available audio channels embedded into the first three video streams, which we then de-embedded in London, and created the surround sound mix from.” This “almost guaranteed that the video was in sync with the audio, because there’s nothing to indicate that the paths are ever in phase with each other.”
For the US Open, there was the added complexity that the vision of a guest might be on one path, but their audio on another, which could have caused problems. “As we went live, it became very clear that the low-latency IP paths were coming back to us a lot quicker than the unilateral world feeds.
“This made cutting around the outside courts for the first few days a little tricky, but the production team soon built trust in the technology, and when the situation arose where we wanted to simulcast another game, from another court, on another channel, whilst still live with our main coverage, the technology allowed the different disciplines to rise to the occasion. Having the added flexibility of being based in a control room in our main broadcast centre, but with the flexibility and additional resources you’d expect from being on site, made this possible,” said Craggs.
World Ping Pong: MADI over Ravenna
Most recently Sky wanted to do full IP production and delivery of an event, choosing the 2016 World Championships of Ping Pong. A key difference here was the audio, which needed to be handled as a separate entity from the video.
“For this we chose MADI over Ravenna. This would not only allow easy integration with the audio networks at both ends, but would provide a phase-coherent path between the two,” said Craggs. “This, more importantly, would allow the sound supervisor to have access to every audio channel all the time, allowing him to pre-set any EQs or delays as required. But, by doing it this way, we faced the inevitable situation of having to re-sync the MADI audio with the incoming video.
“Even though we were using common transmit/receive hardware across a common network, there was never any guarantee that the paths would arrive back at base in sync.” During the first facilities check, they ran a common test card over all the paths and, with some slight configuration tweaks, managed to hit their video baseline, to which they could then bring the MADI audio up to. “Once we hit this baseline, the system seemingly held there,” said Craggs.
As the host broadcaster, there was also an obligation to contribute to the house PA back at Alexandra Palace (North London), but the two-way nature of IP allowed it send back 24 tracks of audio, as well as several vision circuits for use on the monitors above the main table.
“Apart from a few ad-hoc installations of IP transport hardware, nothing else has really changed in terms of workflow,” said Craggs, who believes that anyone who walked in to the gallery during the coverage would have thought the tournament was taking place in the studio next door.
The three-day event, covered 111 matches with seven cameras (five around the main table), with an RF camera moving between the eight outside tables, and a locked off wide view for overall perspective. Camera engineering was on site, and there was a sound supervisor to look after the feeds both ways.
“Camera return operated as normal, the tallies all worked, and most importantly, without any discernable delay,” explained Andrew Finn, Senior Director, Sky Sports. “There was no studio on site. Instead, we hosted from the main TV table. We do this on a lot of programmes now, as it not only saves a fortune on facilities, but it looks and feels far more inclusive of the event,” said Finn.
Director Andrew Finn revealed what happened when all the feeds went down for the Ping Pong World Championships
Everything worked well on Friday and Saturday evenings, until the final match on Saturday night, when everything went black. “All the feeds went down. If this was a traditional OB, and we were on site, broadcasting back to Sky, it’s likely that we would have lost the transmission path for many minutes, and the viewer’s signal would have been interrupted,” said Finn. “However, with the redundancy measures that we’d put in place, which included two satellite backup feeds, carrying cameras one and two, and commentary, I was able to cover the match to its conclusion before we eventually got off air without really any noticeable interruption.” It turned out that it was a telco problem in Leeds rather than any systematic failure on site.
Finn didn’t notice “anything different in the gallery working in that fashion than if I’d been on site, and I think that’s the same for everyone else on the crew. I did spend a little bit more time in pre-production talking to the heads of department. This was as much about reassuring them that our approach to the workflows wouldn’t be affected as it was about discussing the production in detail.
“The executive producer, however, was in a far better position than normal, instead of being stuck behind a wall of monitors in an OB truck or a gallery, he was able to freely move around the venue, confident that he could communicate with all the crew and monitor the transmission on his iPad.
“Overall, the production was a huge success backed up by really strong viewing figures and a very, very positive response on social media,” said Finn.
What about Ultra HD?
With the bandwidth requirements for UHD, will IP work as smoothly, or will it limit the number of streams going back and forth?, asked an audience member.
“The bandwidth requirements are limited across the UK and across Sky Sports,” said Clement, who added that in any case IP wouldn’t’ be appropriate to do on all the events it covers.
“We have to pick off the more straightforward events for which we can apply this technology. The technology would cost us too much. So, we’ve taken a path using sports that we can actually attack at the moment, and had a go at those. The UHD ones are unknown. The bandwidth is going to be considerable,” but as the technology progresses it may be possible to squeeze it down the same pipes we use now, or the pipes will get even wider. Besides, Sky hasn’t been interested in doing the sorts of big events that would be the first to go UHD, remotely — even in HD yet. As the technology improves Clement is sure they will look at, but “we’re not anywhere near that yet.”
Ruddell agreed. “You pick off the easy things first. The communications are the most important bit, then you move to more complex productions as you can.” But “it doesn’t mean move to more and more IP.” It has to be appropriate and the risk shouldn’t outweigh the benefit.
In the US, the drive to remote production is partly due to the huge distances involved, often for third-tier events, where there is a big saving in having a small van with just the cameras on site. In the UK and Europe there isn’t that necessity, and OB companies “prefer to give us a full truck with only three people in it,” rather than building a small van that is only used by Sky’s remote productions, as “they can’t use it for anything else,” said Ruddell.
Besides, “it’s not all about saving money. If you lose your network back to Sky, that’s your lot,” but if you share the production between the venue and the gallery, there is a fall back.
But, to get it right, he believes it will be important to standardise how we do IP productions, “so that we know what IP connectivity to expect” on site.
An attendee asked about all-IP studio operation, but today Sky has a number of advanced, SDI-based studios that are only a few years old, so “what would the benefit be for me of ripping out that few million pounds investment in a studio and replacing it with something new just because it was IP,” replied Ruddell.
“If we get to the point when those studios are due for a refresh, if that technology is the right tool in the box at that time when we make that decision, then maybe that is the way we will go.” And, “if I build a studio based on IP, it has to talk to the rest of the organisation, so there is quite a complicated jigsaw puzzle of what gets upgraded, when.”