Precision measurements: Grappling with latency at the WWE

WWE, the hugely popular global wrestling entertainment brand, recently brought its show to London and the O2 Arena. Providing the live television coverage was a considerable challenge.

WWE provided the technical supervision for the production. Each event was cut live, in a set of NEP mobile units, on location at the O2 Arena. WWE’s English commentary team were on site at the arena for the main commentary duties. PSSI/Strategic Television were tasked with the distribution of the produced event signals for worldwide viewing.

At the same time, each camera feed was sent as an ISO file to WWE’s headquarters in Stamford, Connecticut, for subsequent post production and archiving. The distribution feeds of the finished programmes were, of course, duplicated for redundancy over different carriers, so there were a lot of video feeds moving around.

Adding to the that was the complications of producing international versions. WWE fans can be found around the world, and the market is served by localised versions with commentaries in Mandarin, Arabic and many more languages.

These specialist commentators all work from sound booths in Stamford. To enable the remote commentary, there was a live cut routed across the Atlantic, and these international commentators were each sent back to London where the discrete international commentaries were added to the final distribution deliveries for the worldwide broadcasts.

With multiple audio and video signals over the 5,500km between London and Stamford and back, as well as onwards to broadcast partners worldwide, there was one very obvious significant challenge for the WWE/PSSI teams; minimising latency and ensuring that all the signals stay in synchronisation. The signals can be delivered via Satellite, Fiber J2K, Fiber ASI, IP across multiple vendors and types.

PSSI has been a long time user of Hitomi for lip-sync measurements. “I think our first box was serial number 25,” said Tracy Michaels, director of project engineering for PSSI. Indeed, they acquired that first system, ten years ago, for WWE.

One of the side benefits of travelling to London was that Michaels and his team could meet the engineers from Hitomi and upgrade to the latest units. “Matchbox is a tool that we really cannot do without today,” Michaels said. “We have many clients that use this, and specifically come to us asking for lip-sync readers and analysis.”

One of the key benefits in this application is that the current version of Matchbox supports up to 16 audio channels, which is a real bonus when lining up all the various commentaries and language tracks. The audio meters and synchronisation are available as video overlays, which is a very direct way of understanding what is going on.

Concerned about latency

While lip sync is the critical issue that everyone is working towards, broadcasters are increasingly concerned about latency. As more processing is added, particularly in the video signal, delays mount up and the ‘live’ signal becomes less and less live. And of course, it is different latencies in signal paths that are a major contributor to lip-sync errors.

“Latency is always more pronounced in the IP world,” according to PSSI’s Michaels. “But as remote production becomes more common, it is valuable to know what that latency time is, back from the field, and how you account for it.

“If you know that your latency is 700 milliseconds from origination to your production facility, then you can adjust your production around it, how all the elements come together.”

Discussing remote production, he talked about the multiple paths that signals will take between origin and destination. “When you are talking 22 cameras, they may not be all joined in one transport path,” Michaels said. “There could be multiple satellite carriers, or different fibre lengths, especially with geographically redundant streams. And of course, everything needs to be backed up.

“There’s going to be variations in delay. How many facilities does a stream pass through? How many points touch the stream in the transport? You may find that fibre is very fast, but there is still inherent delay – maybe 10, 20, 30 milliseconds each time – and it adds up, and it will affect things.”

Precision measurements

“Setting up for WWE, we use Hitomi MatchBox to test all our transmission paths, to make sure we have a consistent lip-sync across the board,” Michaels said. “Then, with our experience in the industry, we can spot things like an embedder still running an offset from a previous show. My job is to take a reading, compare that reading with the production trucks readings and that of Stamford’s for each and every feed, then talk with the technical staff so we can pinpoint and isolate each delay.

“A lot of networks say plus or minus 10 milliseconds is acceptable. We like to get it dialled down to zero.” These checks consider the full signal workflows for routers, switchers, audio consoles, tape playback, frame-syncs, multi-viewers, encoders, and decoders.

Marty Pingree, SVP of event technical operations at WWE, summarised the situation. “With WWE, it’s all about the fans. We want them to be able to watch our broadcast without any interruptions, with the best quality so they can see and hear what we have to offer.

“What our performers do in the ring, there are a lot of big hits, and the fans expect to see and hear them at the same instant,” he added. “You don’t have to be a technical genius to know when something is out of sync. Hitomi MatchBox ensures that all our signals are lined up.

“Sitting here, in the truck in London, we have a monitor wall that shows us all our paths that go out to all the services taking the programme,” he concluded. “Thanks to Hitomi, we know that every path has been evaluated and corrected, so the signal leaving us and going to our partners is as good as it gets.”