Royal wedding fans get machine learning-powered ‘Who’s Who’ app from Sky and AWS Elemental

Machine learning is taking a front row seat at the wedding of Prince Harry and Meghan Markle on 19 May with a ‘Who’s Who’ app for celebrity fans. Sky News is teaming with Amazon Web Services (AWS) plus AWS partners, GrayMeta, a data company that powers automated metadata collection, and UI Centric, a creative engineering business, to enhance the user experience of this live-streamed news event using machine learning at scale to a potential audience of 23 million Sky viewers.

Keith Wymbs, chief marketing officer at AWS Elemental, told SVG Europe this use of machine learning will go far: “I really think this is one of those examples of technology that really takes the imagination. It has gone from a manually-entered version in a government environment, identifying speakers in Congress, to celebrities. I can’t say where it’s going to go, but I can see it in sports events, especially where you are tracking people, like cycling or triathlons; you could use it to see how everyone is doing, not just the top five.

“Its use is based on the imagination of the broadcasters because we’re taking that undifferentiated heavy lifting away from them, and putting it into the cloud,” he continued.

For AWS, Wymbs stated, the goal is taking the everyday issues away from the customer, such as how to get feed from A to B, to allow them to use their imagination where it counts; bettering the end user experience of the service. “We’re taking video delivery in the cloud end to end, adding image recognition capability, and enhancing it with the help of partners on metadata and the user interface. We’re pulling a lot of things together. Amazon is a provider of building blocks to help media companies provide better experiences for their users; we take care of the core video flow, and they can take care of the user experience.”

Wymbs said the joint effort is the result of a long collaboration with Sky. “We’ve had a long relationship with Sky, which was actually an original investor in Elemental, in its D-round of funding. Also, we pre-briefed Sky when we announced Media Services, and the result is this project which is a really good example of Sky thinking about how they can do something experimental and improve the end user experience with the cloud.”

Sky News will use pioneering cloud-based machine learning and media services to name guests as they enter the chapel, and enrich the video content with facts about the attendees. Cloud is really coming of age, continued Wymbs: “Over the last five or six years, companies like Sky have begun to trust the cloud. In that time period, we have been seeing a lot of companies pushing the envelope in the cloud, and seeing broadcasters starting to use it more. The BBC was very early on in its use of the cloud, which it started using in 2012, and we are now seeing others take it up as well.”

Users of the Sky News app will be able to activate the app’s ‘Royal Wedding: Who’s Who Live’ functionality to automatically highlight celebrities as they appear on screen.

Importantly, this is a second experience for viewers, stated Wymbs: “The camera feed we’re using is a separate feed from the TV broadcast; it’s a red carpet feed of celebrities entering the chapel. Also, if a viewer joins the stream mid-way and they catch a celebrity mid-promenade, they can simply click on the celebrity’s name in the app and the feed will go back to the start of where they showed up in the stream; part of the metadata is a timecode that’s generated live. It’s pretty cool!”

Celebs will be annotated with details such as their names and career highlights, plus their connection to the royal couple, using on-screen captions and graphics. Users can navigate the celebrity data without leaving the app and keep the primary video content on-screen, for a self-guided, hands-on experience for every celebrity sighting.

A video feed from an outside broadcast van located near St. George’s Chapel will capture faces of arriving guests and feed the signal to an AWS Elemental Live small form factor appliance located nearby for real time ingest to an entirely cloud-based workflow.

Video processing is handled with the AWS Elemental MediaLive service. AWS Elemental MediaPackage integrates digital rights management for secured distribution over the Amazon CloudFront content delivery network (CDN). Sky News will also use the AWS Elemental MediaPackage service for live-to-VOD applications such as catch-up TV.

In parallel, Sky News is combining the GrayMeta data analysis platform with the Amazon Rekognition video and image analysis service for real-time identification of celebrities and tagging with related information.

The Amazon Rekognition video and image analysis service allows customers of AWS Elemental to upload 1,000 images per month for just 10 cents. “This gives you an idea of the level of capability you get,” said Wymbs. “It’s a very inexpensive way to explore these kind of opportunities that media companies are wanting to provide to customers.”

Finally, Sky News is using the Amazon CloudFront content delivery network to unify the content for faster distribution to viewers. UI Centric has designed and developed the front-end application and video player to enhance the viewing experience and to leverage the Sky machine-learning workflow.

Video content from the application will also be made available on demand after the event.

“Sky continuously searches for ways to innovate and bring better coverage to its customers. This new functionality allows Royal Wedding viewers greater insight into one of the biggest live events of the year, wherever they are. We’re excited by the software’s potential and ability to give audiences new ways of consuming content,” said David Gibbs, Sky’s director of digital sports and news products.

On TV, Sky News will be the only broadcaster to cover the Royal Wedding on 19 May exclusively live in Ultra HD. The broadcast will be fronted by Kay Burley from special studio locations in Windsor covering everything from guest arrivals and the bride’s entrance to the couple’s nuptials in St. George’s Chapel at midday and the subsequent carriage procession.

Arrivals for the Royal Wedding will begin at 9:30 a.m. GMT and continue until the ceremony begins at 12:00 p.m. GMT. The Royal Wedding: Who’s Who functionality is accessible at Sky News online or via the Sky News app for mobile devices