Remote production in action: Transporting the Finnish Ice Hockey League

The Finnish ice hockey season went on air in September via a production hub in Helsinki. SVG Europe talked to Broadcast Solutions project engineer Andreas Dyballa about how this landmark project came to fruition.

“Ice hockey is the prime sports product here in Finland, along with Formula 1,” said Dyballa. “It is like Premier League or the Bundesliga. The furthest ice hockey stadium is in Oulu, which is about 700 kilometres from Helsinki. It’s just a little bit below the Arctic Circle!” he said.

The Finnish league plays more than 450 matches per season, with up to seven games simultaneously. Covering all matches simultaneously (in both HD and UHD) and adding studio production required production company Streamteam Nordic to build a centralised production hub. The technical solution was designed and implemented by Broadcast Solutions.

The league is broadcast by Finnish/Swedish telecom provider Telia via its IPTV service, set-top boxes and streaming platform, beginning with the 2018/2019 season.

The new facility contains eight production rooms, an audio control room, five off-tube rooms and an MCR connected to seven remote sets. All venues are connected to the Helsinki hub via Telia´s DWDM (Dense Wavelength Division Multiplexing) network. The virtual studio, which produces pre- and post-game live discussions, is linked to the hub by four dark fibre lines.

Streamteam’s production facility also includes a centralised video referee room with up to four video referees overseeing the matches in all venues. The overall system design takes a hybrid approach, leveraging IP-workflows for interconnecting the venues and hub, while production within the hub and the studio uses 3G baseband format. Transport between the hub and the venues is based on Nevion’s Virtuoso platform, with JPEG2000 over 10 Gbit fibre connections.

Currently, two of the seven matches are produced in UHD and five in HD. For the UHD matches, a large remote set is used with 14 Panasonic cameras. The HD matches are covered with a remote set consisting of 9 Panasonic cameras.

Four cameras, also Panasonic, including two on telematics robotics are on location in the virtual studio. Within the hub, a Grass Valley Momentum production platform is partnered with seven Kahuna production switchers and a Sirius 850 video matrix (456×840).

Supporting each camera signal’s replay and ingest, 13 Grass Valley LiveTouch servers provide 90 ingest and 16 replay channels. Grass Valley’s FileFlow and Momentum systems handle the file and metadata transfer between the LiveTouch servers and Telia´s OTT platform. Lawo VSM is used for overall control of all video and audio signals for the router, vision mixer, multiviewers and tally.

“I was the system architect, involved from the first discussions with Streamteam about this project,” said Dyballa. “I created the system solution and then on the project I shared the management and engineering planning with my colleague Yriö Pananen. He took care of the detailed audio and video planning and I concentrated more on the transport solution and the control system.

“Telia is not only the rights owner and content distributor – it is a big telco company. It owns a nationwide network so would have been able to create its own IP network for this ice hockey project. I don’t know the reason why they did not,” he said.

“Right from the beginning, they said they would not provide an IP network but they would provide, per stadium, one light beam wavelength on their fibre optic network. Not like a dark fibre; it’s just one wavelength on their DWDM network. How the potential production company would utilise this one wavelength (internally we call it a ‘grey fibre’) would be up to them.

“From each stadium Telia provides a fibre injection point at a standard wavelength and then we receive on the other side here in Helsinki at the production hub. From a production standpoint this is like a transparent network; we don’t know what happens in between.

“At the beginning we wondered how we could utilise this one wavelength. Of course there are some really expensive telco systems in which you can squeeze 100GB though one of these wavelength devices. But that was out of our scope.

“The only other option was with a 10GB network providing sufficient bandwidth: this is not transporting uncompressed but you could utilise a no-latency compression like J2K (JPEG2000), or maybe even TICO or VC-2. But in the end VC-2 and TICO do not have enough compression: this depends on how many UHD signals you have to transmit. If everything is UHD the 10GB would not be enough. So it left us with only the J2K compression.

“We use the Nevion Virtuouso platform. For a 3G signal we compress down to 150Mbits per second. My understanding is that J2K is the best compromise when it comes to bandwidth, latency and cost,” said Dyballa.

So on a full match day with seven games, the team produces two of the seven in UHD and the other five in HD? How does that work in practice?

“Yes at the moment we produce two of the seven matches per match day in UHD – but we transport only Camera 1 in UHD. All the other cameras are transported in 3G; but as you know 80 per cent of on-screen airtime is Camera 1 anyway.

“We have five so-called small remote sets with nine cameras, and two bigger remote sets with 14 cameras. Out of those nine cameras, five are CCU full broadcast-style and four are PTZ fixed installed above the goals, which are mainly used for the video referee system.

“So we transport in total five times nine which is 45, and then two times 14 which is 28. That’s 73 cameras in total, plus Camera 1 in UHD from each remote set. In total on the transport system we have 117 transport channels between the remote sets and the Helsinki hub.

“We have one Nevion Virtuoso unit at each of the remote sets. They convert from SDI to JPEG2000, and then Telia sends it on a redundant fibre path on its network – they call it ‘red and green’. Here in Helsinki we have the hitless merge (SMPTE 2022-7) automatic changeover, which means if one of the fibre lines fails you will receive the data immediately from the other side. It’s a proven SMPTE mechanism.

“From the transport point of view, with the Nevion Virtuoso platform, honestly it was the right decision and I am extremely happy. It runs absolutely stable. And the support from Nevion has been very good throughout.”

On the production side the team decided to go for Grass Valley LiveTouch rather than any other replay server. What was the decision-making process there?

“When the Grass Valley guys came to show us the LiveTouch last year it was rather new. They were convincing about their overall concept. And when you sell five Kahuna mainframes and a big Sirius routing matrix, with LiveTouch you get a better price! It was a big package in the end.

“The SAM guys, as they were at the time, were very interested to get LiveTouch into such a big installation. And on the other side, when the guys from Streamteam saw LiveTouch they wanted it.

“Regarding EVS, at the time it was not clear to what extent they could ingest and replay UHD and how many channels we would do in 3G. This was one and a half years ago – at which time the LiveTouch was the only hybrid system where you could ingest 3G or UHD in one server and have a 3G or UHD replay.”

So it is a remote production hub. How much is still done in trucks at the match venues?

“We don’t refer to trucks, just remote sets – because there is no video mixing on-site. Of course there is CCU in the truck. The shader and video engineer are in Helsinki but the CCU itself is sitting in the truck.

“The CCU has an SDI output, because we could not get the camera output directly in IP to Helsinki. That would be possible with the Grass Valley cameras, which have direct IP, but the customer decided to use Panasonic cameras.

How many people remain on-site, helping to produce each match?

“There are five CCU-operated cameras with five operators, plus one driver/engineer who drives and does the basic set-up. With remote production you set it up so that you basically plug and play. You set up the camera, switch it on, connect the truck to Telia’s fibre interface and everything should be available. So far, that works fine.

“Yes we are doing remote production for real, every week! The last thing I would be waiting for – which would make the system integration much easier – is if you could transmit the camera signal over IP and have the CCU here in Helsinki. That would make it so much easier when it comes to tally, intercom and feeds, which you still now have to transport over the 10GB link. That is the last piece.

“We decided to go for an SDI-based environment, with 450 inputs and 820 outputs. The reason was very basic: IP was more expensive than SDI. With a smaller scale and a little bit more time I would go for the IP environment. But there’s a breakeven point: with all the gateways you need you’re still cheaper with SDI,” said Dyballa.

Broadcast Solutions also designed the league’s VAR set-up, based on Simplylive’s Ref&Box system and ViBox super motion servers with MMR-110 USB recorders. What is the workflow there?

“Here in Helsinki we have seven Simplylive servers, one for each match. Each server ingests eight feeds – so not all of the 14. They assign the important cameras for the referee, via the matrix.

“Then there is a ‘league situation room’ here in the hub, with four operating stations for the referees. For a full match day, three of the referees have to cover two matches. That actually works fine, using multiviewers. If a situation happens in game A, the referee simply connects with his client to the server for game A and then connects with the referee at the game to discuss the situation via the Riedel intercom system,” said Dyballa.

“The referee then refers to another Simplylive client via a normal laptop on the sideline, seeing exactly the same picture as the Helsinki referee. Via intercom they can discuss the situation and then make a decision.”