Live from Amsterdam: Ajax steps into the future with AI-driven player analysis and live streaming

At the UEFA Youth League Ajax U-19 v Bayern U-19 match on Wednesday, Ajax FC Training Academy hosted a live demonstration using AI-driven live broadcast capture for player analysis and simultaneously for Ajax TV.

Ajax FC technology consultant Max Reckers (L) and Mobile Viewpoint MD Michel Bais before Ajax U-19 v Bayern U-19 UEFA Youth League match in Amsterdam on 12 December

The Innovation Team at Ajax partnered with Hikvision, Mobile Viewpoint and TNO for this technology collaboration. Dutch research foundation TNO and Chinese video surveillance company Hikvision have developed an ecosystem for live sports production and analysis using new technologies to automate the capture of live sports, based on AI algorithms. The four-lens 180-degree Hikvision camera used for Wednesday’s demo is the company’s very first unit in Europe.

The Ajax Training Academy, led by Ajax Innovation technology consultant Max Reckers, has taken the lead on this AI-based project – which from Wednesday’s UEFA Youth League match has also now been adopted by Ajax TV for live streaming.

Mobile Viewpoint has worked for the last nine months on productising part of this ecosystem to make it ready for broadcasters and editors to be used in online streaming and live television.

TNO’s technology focuses on tracking players based on AI and low-cost hardware; recognising situations and persons with very high accuracy, including behavioural detection and tracking; and machine learning to recognise patterns and act upon that in terms of direction.

Mobile Viewpoint’s IQ Sports Producer is an AI-driven production suite that supports the capture and delivery of live

Ajax U-19 v Bayern U-19: Bayern ran out 2-1 winners

sports streaming from any location. The combination of Hikvision’s four-lens camera with Mobile Viewpoint’s AI technology captures matches by tracking players, balls or other objects and live streams the resulting content – removing the need for production facilities, camera crews or directors.

The IQ Sports Producer platform features the 24MP 180-degree camera, covering the entire playing field; a 2U MVP PC box with IQSP software and HEVC encoding; and an IQSP LinkMatrix online management dashboard. The system’s de-warping and stitching technology provides a corrected image with straight line markings. A pan and zoomed image is automatically created using specially-developed game-tracking technology.

Mobile Viewpoint managing director Michel Bais said, “smaller or lower leagues of sports can now get a chance to live stream their games. You already see this happening with success using image recognition to follow the game using a single camera.

“AI will make these productions more interesting because it will enable goal detection, yellow and red cards, corners, time outs or injuries. Together with player analysis it is possible to create automatic summaries which will appeal to many more viewers compared to the complete game. Also, AI-based advertisement control will make those games more interesting for sponsors.”

IQ Sports Producer, first unveiled at NAB in April, is now with Ajax and Feyenoord in the Netherlands along with other beta customers internationally. Bais said “recognition-based AI technology was expensive in the past but is now quite cheap to get. As sports viewers move online, second and third leagues can now reach an audience, especially as team and club-based viewing is growing. Meanwhile, new streaming channels are looking for alternative quality sports capture and a return for their sponsors.”

As the partners continue to work on the system, Bais said they realise that “the cameras need protection against the weather, and higher resolution cameras are required for larger fields. AI is the future but needs a lot of training, as ball detection is sharply context-sensitive. AI algorithms are defined on a single image basis; there is no history context, as there is in video encoding — you need to develop your own training data.”

On the road map for the future? Better ball detection; multicamera auto director, delivering 2 x 4K at 16m; automatic replays after goals; automatic summary; and AI-controlled ad insertion.

Bais continued, “Ajax and TNO have been working on player tracking and AI for a long time. We got involved where we said to TNO ‘let’s make a deal together where we adapt your technology into a real product that also fits the requirements of our customers, with the ability to add overlay and advertisements and those commercial things’.

“We started in April at NAB, with an example of what we could do. The surprising thing for us was that at NAB the most interest for anything on our whole booth was this product.

“People said ‘but how do you deal with the connectivity issues?’ We said, ‘that’s what we do, we’re a connectivity company’.

“We’ve spent the last months bringing it all together in one hardware box, including bringing the algorithms back down on Nvidia-based hardware. We had to make it all smaller and more affordable.

“Now we think we are ready to roll out. With sports like ice hockey and basketball, where the players largely stay quite closely grouped together, it can work really well where it looks almost like what you would see on television.

“When 5G [contribution] is introduced,” said Bais, “you could bring the box away from the customer – you’ll only need the camera and you just bring the IP from the camera back to the cloud. You’ll have so much high bandwidth that you won’t need hardware on-site any more. You just put the camera up and away you go. Next year will see a lot of 5G trials, and we expect it to roll out [properly] in 2020.”