Can artificial intelligence do the job of an audio engineer?

By Michel Bais, managing director, Mobile Viewpoint

Artificial intelligence (AI) is beginning to be adopted across the broadcast arena as a new initiative to enable productions and operational processes that were previously cost prohibitive or expensive. It is not positioned to replace jobs, but to enable productions that were previously just not possible. But can AI be applied to the audio?

As a video technology company, we often get wrongly accused of not prioritising the audio element as much as the video quality. We have been implementing AI for video as an alternative to camera people so it may be interesting to some that as a video technology company, we are developing what we believe is a world first: a method to also capture audio as part of the production, but without the need of an audio engineer.

But what is this exactly?  One example is in sports production where a panoramic camera is utilised above the field of play and captures the whole pitch in super high resolution. Then, using the power of AI, a virtual cutout of the action in full HD is created which live streams only the ball and the action of the players around the ball. There isn’t a cameraperson or director in sight and it enables live sports production at a fraction of the traditional cost. This could be ideal for those games that previously could not be live streamed, such as youth sports, training games and grassroots games, and it allows sports clubs and media right holders effective ways to monetise or promote their content that were previously not possible.

AI audio

But what about the audio element? A major part of any sports production is capturing the mood of the game, of which all the sounds and noise are a fundamental part of the experience. Traditionally there would be a bank of microphones on the pitch-side, capturing different parts of the pitch and the reaction of the spectators. So, if a goal is scored by the home supporters than the reaction of the home supporters can be ingested into the production. Any swearing or bad language can be taken out. It is the job of the audio engineer to do this.

As part of an AI production where AI creates a virtual camera to track the ball, we believe it would be interesting to also allow the AI to follow the audio in the same way, taking the place of the audio engineer and giving a much more varied and compelling audio experience. This we feel is a world first. In reality this means developing a panoramic audio capturing device which replaces the bank of costly microphones. The AI will only reveal the audio that is relevant to the video shot and embed that with the live stream.

Wide appeal

It is something under serious development at IQ Video Solutions.  The future of this technology could be applied to many situations. At some point for budget productions, we see the panoramic camera being used as a replacement for a multicamera production, whether that be in a panel show, a council meeting or a studio set up. If the audio capture device can filter out other noises and only pick up the relevant audio to that specific shot, then we can create much more relevant productions and reduce costs accordingly.

Are we trying to replace jobs with AI? First, we remove the need for the camera person and director, and now the audio engineer? This is not our intention – the AI platform is designed to enable productions that previously were not feasible by not having the budget or resource to create a professional production. The AI Audio Engineer is an extension of that philosophy and adding to the overall AI proposition. There will continue to be a place for manned productions for those who have the budget.