Right place, right time: Sony Sports Innovation on the development of Hawk Replay and skeletal tracking

Hawk-Eye’s Every Shot, Every Hole in use at the Augusta Masters 2019

In 2017 Sony created a new internal division, Sony Sports Innovation Group. This encompassed its 2011 acquisitions, the Hawk-Eye Innovations and Pulselive subsidiaries, as well as the Sony Professional production services business, providing production facilities at the time to deliver all live content for the FIFA World Cup 2018 from all venues as general contractor.

While its Media Solutions arm sells kit to broadcasters and content players, the Sports Innovation Group deals more with leagues and clubs, although there is crossover between the two depending on client needs.

In July 2017, David McIntosh was appointed to CFO of Sony Sports Innovation. Working across 20 sports, delivering services in 90 countries at 11,000 events in 500 stadiums, Sony Sports Innovation is at the very heart of sports officiating as well as delivering digital solutions for the world’s largest sports federations, leagues and clubs through the creation, development and management of digital platforms, reaching millions of sports fans every day.

Harnessing Sony’s leadership in live production and sports technology, this fast-growing business group, including more than 400 staff, is creating transformative new experiences for fans, sporting organisations and media rights holders.

Aggressive sports growth

The plan behind the sports division is to get the best out of the development that goes on in Hawk-Eye Innovations and Pulselive, says McIntosh, who is also CFO at both subsidiaries. He adds: “The point of Sony Sports Innovation is really to use that Sony platform to aggressively grow our participation in sports, in sports production and even potentially in sports entertainment, if that’s the direction that we want to go in.”

During 2020 Sony has fallen back on its strong market position to enable it to keep moving forwards, says McIntosh: “A lot of difficulty has come to sports production over the COVID period. Going back to March time, we’re a global business and we were starting to look at China closing down and then other countries following suit, which was quite a frightening experience because your businesses is effectively drying up and you’re watching it disappear night after night or day after day.

“But I think the benefit that we’ve had is that by being part of the Sony Group, we’re still financially very strong and I think that’s given us the ability to make better decisions.”

Ed Hawke, product director at Hawk-Eye (and with a purely coincidentally matching name), adds: “Our business [mostly] disappeared overnight [when live sport ceased], and yet we were able to continue working. We were able to keep all our staff employed.”

Right place, right time

Hawk-Eye has been working on the further development of the Emmy Award Winning Every Shot, Every Hole project, a remote clipping product which it first implemented with CBS and Thumbwar at the March 2019 Augusta Masters.

Says Hawke: “The things we’ve been pushing forward on this year luckily span the range of our core technical capabilities.”

Every Shot, Every Hole allowed operators who were offsite in Basingstoke in the UK and in Atlanta, Georgia, US, to clip every single shot played during that tournament, resulting in 22,000 shots over four days which were then uploaded to masters.com.

Adds Hawke: “I think we’d kind of seen the shift in the industry towards trying to reduce the number of people on site [when we developed that product]. And that’s obviously been going on for a few years now, for cost and carbon footprint reasons. People are trying to minimise the number of people they need to deliver live sports.”

The next innovation after Every Shot, Every Hole was Every Shot Live, which moved on to provide a live stream of every single golfer. It was used in March this year at The Players Championship in Florida, US, for just one day before the tournament was called off due to the pandemic.

Notes Hawke: “It was from that point, really, that we started to pivot that [product] into an instant replay tool for the replay operators.

The Every Shot Live operational room based in Basingstoke, UK

“And so we took the model of Every Shot, Every Hole and then looked at expanding the functionality of that so that we can provide a fully remote broadcast replay tool. This new product is called Hawk Replay. That allows replay operators to sit at home in their living room, attired however they desire, delivering instant replay to live sports broadcasts this year. We have been providing that to a couple of major US broadcasters this year, for sports like golf; it’s been a case of the right product at the right time.”

He adds that the development of this product was driven by engagement from Hawk-Eye’s partners who had been using the company for golf broadcasts. Hawke states: “They were kind of excited about the idea of that product. So discussions were going on between us and that’s when we started to invest time into [Hawk Replay].

“Having the break from [live sport because of] coronavirus allowed us to start focusing on that [development] full time,” Hawke continues. “As sport has started to come back, I think that’s actually been really important for the broadcasters as they didn’t have to reduce the quality or scope of the production despite needing to keep the number of people onsite to a minimum, because they’ve got guys sitting at home in Florida providing replays for golf tournaments all around the US using our technology.”

Hawke adds it was a case of kismet on the technology side, but also a recognition of where broadcasters were looking to go that meant Hawk Replay was the right product for 2020.

He notes: “We knew from prior to coronavirus ever being a thing that the REMI model was becoming more and more ubiquitous; there were lots more people looking at using that going forward so we wanted to take that one stage further and be able to provide these sorts of services. Every Shot, Every Hole wouldn’t have been possible [this year] if you’d had to have 40 operators suddenly descend on an already crowded site. This was the right product at the right time for the pandemic in being able to add that benefit for broadcasters of being able to use it from people’s homes.”

Skeletal tracking

Another area that Hawk-Eye has been looking at over the lockdown period this year is skeletal tracking, moving further up the ladder from traditional player-tracking technologies, which use centre of mass tracking. The latter picks a point per player, showing the image moving around in 2D. Skeletal tracking allows for more realistic and more accurate 3D imaging.

The skeletal tracking development scheme stems from a project with FIFA, which asked Hawk-Eye to look into the area nearly two years ago.

Hawke explains: “At the end of 2018 we started embarking on a skeletal tracking project. It’s driven by FIFA who stated after the 2018 World Cup that at the next World Cup in Qatar, they wanted an automated offside product. Being that we were FIFA’s partner to provide VAR at the 2018 World Cup, that was something that we really wanted to be at the forefront of in terms of developing products and services.”

Hawk-Eye realised that skeletal tracking for 3D imaging would be required to make an automated offside product effective. Hawke continues: “So we started to look at, actually, can we provide this skeletal tracking that you need to provide this auto offside product? Obviously you need to know when a ball is kicked, [but also] you need to know where everyone’s limbs are to be able to provide an accurate offside line. So we kicked that off then, but more recently we started to realise that rather than just an auto-offside product, there are loads of use cases for skeletal tracking outside of that.”

This year, Hawk-Eye has debuted that technology alongside Major League Baseball (MLB) in the US as part of their Statcast product, enabling it to go live at 30 ballparks across the US. MLB has recently finished its season and the product has been positively received by the Federation and the teams, says Hawke, because of its ability to provide high quality and high accuracy of data, and also because of the volume of data it provides. The latter allows MLB and its associated teams to start looking at areas such as biomechanics of players.

Adds Hawke: “So they can actually get real-time feedback from matches on those sorts of elements, like throwing actions, that previously they’ve had to go into labs to be able to measure. It’s a product that we’re looking to roll out, [and are] actively working with stakeholders and a whole ton of sports on that.”

Another area that skeletal tracking feeds into and that has become clearer to Hawk-Eye over lockdown is the engagement of broadcasters and others in virtual reproduction. Explains Hawke: “I think virtual reproduction is probably one of the more exciting aspects of the skeletal tracking, outside of the people who, like myself, who can geek out on the data. Actually being able to fully reconstruct sports digitally in 3D is something that can provide benefits to broadcasters in being able to help the storytelling to the fans.”

Sports broadcasting in 2021

Looking forward to next year, McIntosh says 2021 is going to involve even more change, including more remote working and the growth of UHD HDR. He adds: “We’ll see the continuation of change because I think what has happened during the past year – or a little bit longer through COVID – is we have come into a situation where people are accepting change much more readily and much faster, and I think that will definitely continue. Whereas previously people would have said, ‘Oh, I’m not sure about that’, [after 2020] they’ll be much more open to saying, ‘Let’s do something different, let’s push it forward and let’s make that change’.”

Hawke adds: “One of the changes that has started to come in now, but I think will become more ubiquitous over the next year, is the change towards being able to use 5G, so increased connectivity everywhere. I think that will become something that starts to drive change in the industry, both from cloud platforms to do things on – so whether that’s virtualised production as complete virtualised production or whether that’s just cloud-based media asset management, or cloud-based editing.

“I think all of those things will start to become a lot more commonplace in the industry, but then also leveraging 5G instead of other technologies for more broadcast-oriented things like streaming video back from cameras, for example. I think we’ve started to see those firsts happen over the past six months, and that’ll continue to happen and become more commonplace next year.”

McIntosh adds that one more area that has come to the fore in 2020 is staff welfare. “When we look back at the past year, when COVID struck, we had hundreds of people all across the globe [and we had to get] them back and then set up new safe environments. Now, in the COVID situation, staff welfare has become much more important. I think some of us, like me, for example, look back and think, why was it I spent so much time on an airplane and sitting in an airport lounge, what was I actually doing? Staff welfare has been a real concern for us over the past six or nine months and I think that will continue to be more important to businesses, but also be more important to employees in terms of expecting a different level of support.”

On how Sony Sports Innovations is going to develop over the next six months, McIntosh says: “The next six months is about working out how things are really going to restart because we’re not going to go back to how it was before. Each federation is looking differently at how they’re going to restart; they will each come back in a different way [that is most appropriate to their sport].

“I think for us, it’s about working out with those sports federations how we can help them to get back and help them to be not just competing, but doing it in a safe way. We’re still absolutely in midst of change. We need to work out federation by federation and sport by sport how to make that return most relevant for each one. And that’s where, particularly with Hawk-Eye, we’re so lucky, because I think we’re now dealing with 24 of the top 26 major sports worldwide. That’s where our challenge is; how do we actually help all of those sports?”