Riot Games deploys longest at-home production to date at LoL mid-season invitational in Asia
The eSports team produced the event out of its LA studios
Few organisations in the live production business are pushing the envelope in at-home production the way Riot Games’ eSports team does. Building on groundbreaking at-home shows from the 2018 League of Legends World Championship and Mid-Season Invitational, Riot Games upped the ante during May’s 2019 Mid-Season Invitational (MSI), which took place in Vietnam and Taiwan.
“This is the longest remote production we have ever done and definitely one of the longest anyone has ever done,” said Mitch Rosenthal, head of production and operations, eSports, Riot Games. “This was by far the most challenging remote production that we’ve ever done as an organisation, hands down.
“But, at the end of the day, viewers at home had no idea, and players at the event had a great experience. It took a lot of big brain power — both internally and externally — to make it happen, but, going forward, there’s not a place in the world that I feel like we can’t [produce a live event].”
In addition to the connectivity challenges presented in a less developed nation like Vietnam, Riot also faced issues in obtaining the infrastructure and equipment needed to pull off one of its largest productions of the year during the Play-In Stage at GG Stadium in Ho Chi Minh City and Group Stage at National Convention Center in Hanoi.
“From a production and operations standpoint, we ran into some unexpected issues in Vietnam, but our team was able to adapt on the fly and figure out some amazing workarounds,” said Rosenthal. “In the end, nobody knew any different except for us, which is one hell of an accomplishment.”
Ho Chi Minh City: Overcoming transmission and onsite challenges
The eSports team leverages the company’s Riot Direct network (the ISP used to service all LoL gameplay) to efficiently send feeds from tournament remotes to its Los Angeles production facility. To use Riot Direct for MSI, the eSports team was forced to devise a roundabout transmission path: from Los Angeles to Japan to Singapore (where Riot Direct PoP is located) and finally to Vietnam.
This more than doubled the latency of Riot’s standard threshold for at-home productions from 180 ms to 300 ms. Riot had three 1-Gbps circuits connecting LA to its operations in Ho Chi Minh City and Hanoi.
The LoL mid-season Invitational Final was held at Heping Basketball Gymnasium in Taipei, Taiwan
Although the Riot team expected new obstacles, its first large event in Ho Chi Minh City proved especially challenging. The team was unexpectedly unable to deliver its normal equipment kit to the city and, as a result, was forced to create a makeshift production/transmission flypack comprising equipment from the less sophisticated internal system used by the LoL Vietnam Championship Series (VCS) operation.
“We really had to get creative and use every single resource at our disposal to put together a kit that would work for us in Ho Chi Minh,” said Matt Donovan, broadcast engineering manager, Riot Games. “It was completely different from our normal standards for an international show, so it was an incredibly heavy lift to make sure that we were able to deliver that same experience to the players and viewers.
“But I think the overall broadcast result for fans and the in-game experience for the players were great. Players were happy, and viewers were happy, and, when it comes down to it, that’s all that matters.”
The team was able to cobble together a flypack based on VCS’s Blackmagic Design 60×60 router and ATEM 2M/E switcher. Riot had six transmit lines to LA (it typically would have 30 for an international event) and two receive lines from the venue. The production team used the ATEM switcher to subswitch the feeds, and the Los Angeles-based TD subrouted the ATEM switcher over a touchscreen connected to the switcher frame in Ho Chi Minh City.
“The TD was essentially precutting 28 cameras/sources down these six lines from the venue, then cutting between the six subswitched sources in the LA control room for air,” said Donovan. “This creative solution ended up being an efficient way to have the same venue I/O the producers were used to without having an equivalent number of transmission paths.”
According to Riot Games technical broadcast manager Michael Caal, a makeshift communications system was created using Unity and Discord to connect LA to Ho Chi Minh City using iPhones and laptops. In addition, multiviewer was eventually encoded to provide the directorial team back home a reliable view of all the cameras and allow them to dynamically choose what to be subswitched down the line. Other lines also served as dynamic routes enabling the EVS team back home to grab feeds needed to build highlights packages.
In addition, Riot IT teams had to find sources for more than 80 PCs and network gear that did not arrive. To run the HCM terminal and game servers, they borrowed the gameplay network stack that player teams were going to use to practice and train at their hotel in Hanoi. Riot Production Management was able to secure practice PCs from a local gaming company to outfit the hotel rooms for the pro teams to practice on.
“I was really impressed with what the local vendors and the [VCS] facility teams were able to pull off, with the small amount of resources that they had,” said Donovan. “It was definitely a challenge, and it was a great display of collaboration and ingenuity between multiple production and logistics teams. But I think the end viewer and the players got the same experience they would have [if we had our full kit]. To pull that off was pretty spectacular from the production-management side, so I was extremely impressed with how it worked out.”
Evolving at-home production
The current at-home solution has been deployed since last year and was engineered to overcome two of Riot’s former at-home system’s technical constraints: bandwidth and circuit reliability. This MSI event was the first real-world opportunity for Riot to produce a show inside the primary limitation, bandwidth.
With only single-gigabit point-to-point links available at each of the three MSI venues (instead of the usual 10-Gb lines required in the past), Riot used Haivision’s low-latency H.264 encoding TS over UDP to produce the same 40 1080p60 video channels and 256 audio channels over just one of the three available 1-Gbps lines. This allowed other venue and production internet traffic to traverse the second line and leave Riot one line for broadcast failover.
“With this successful four-week, three-venue, 1-Gbps at-home production under our belts,” said Donovan, “I am curious when we will get the chance to put our tech to the second REMI-limiting constraint that our system was built for: circuit reliability.”
Hanoi: Balancing resources, continuing the at-home push
After a week of Play-In rounds in Ho Chi Minh City, it was off to Hanoi for the Group Stage, for which Riot had already been loading in throughout the operation in Ho Chi Minh City.
“When we were in Ho Chi Minh City, we were already loading in Hanoi, so we couldn’t have all of our equipment and encoders in Ho Chi Minh City,” Donovan explained. “On international shows like this, we are always trying to find a balance and make the decisions necessary to deliver the highest-quality value with the hardware that we have available.”
Riot continued the at-home workflow in Hanoi, with the bulk of the production centralised in a control room at its LA NALCS studios. All switching, camera shading, and graphics insertion and the observer room were located in LA, along with casters and analysts.
“The way I look at it, we’ve now got the home-run–production [workflow] nailed. Moving forward, it’s more about the degree of difficulty in the regions that we go to around the world,” said Rosenthal. “Connectivity and infrastructure on the ground are two of the major challenges, as well as finding talent locally instead of having to fly people in.”
Read the full story here: https://www.sportsvideo.org/2019/06/28/riot-games-deploys-longest-at-home-production-to-date-at-lol-mid-season-invitational-in-asia/