Live from Rio 2016: NBC Olympic Tech Leaders Reflect on the Massive and Successful Production Effort
The 2016 Rio Olympics have come to a close and the NBC Olympics machine, for the most part, has begun the process of being dismantled (a much smaller technical presence will remain on site for the 2016 Paralympics). The end of the games also begins the process of evaluating what worked, how operations can be improved, and how the entire Olympics production and technical philosophy might change for the 2018 Winter Olympics in PyeongChang, Korea. But the end of the Olympics also provides an opportunity to step back and reflect on an Olympics technical team that gave the production team the content fuel to serve Olympic fans an unprecedented amount of content on an unprecedented amount of platforms and devices.
Dave Mazza says the NBC Olympics production team was able to get to full speed quicker than usual this year.
“I could not have been more proud of all that we as a team accomplished here in Rio and in Stamford, Orlando, and Hialeah,” says Dave Mazza, NBC Sports and NBC Olympics, SVP and CTO. “In the run up to the games, there were obviously tremendous concerns about all sorts of things. But, everybody kept their heads down, pulled together and got their portion of the job done. We may not have been working on our Plan A, Plan B, or even our Plan C, but it is a testament to so many seasoned, flexible and inventive professionals that we able to go up, over, and around most obsticles standing in the way of our success. People asked if we were going to water down our plans in light of the difficulites on the gound. We not only did not do that, but the entire team executed NBC’s most ambitious Olympics coverage plan to date, on all fronts – Broadcast, Cable, Digital, and Social.”
And then there are the freelancers, the ones who volunteer of their own free will to be part of project that every time it occurs is basically producing an event in a city that has no experience doing what they are doing. For all of the veteran experience it is, in many respects, a first-time effort.
“We couldn’t do it without so many freelancers who have worked with us before,” says Dave Mazza, NBC Sports and NBC Olympics, SVP and CTO. “And I guess they like it enough as they come back with a smile on their face and knowing that there are certain things, like transportation, that don’t work well at an event this size. And they also know about the crazy way we have evolved over the years so they can figure out new things without hardly even having to be told. And also this year we have about 15 really good young guys in the mix.”
Adds Terry Adams, NBC Olympics, VP of IBC engineering: “The veterans here are A level and even some of the younger ones who are working their first games had a skill set well beyond what we thought they would have. They were terrific and as far as our staff in the IBC there were no duds.”
The high level of performance also extended out to the venue operations team as Chip Adams, NBC Olympics, VP of venue operations, said the veterans involved in venue operations once again delivered.
“We also found some great new tech managers to roll into places as it was difficult to get some of our regular tech managers in because of the payload that NBC Sports now has between The Open Championship, Sunday Night Football starting soon, etc.”
One of the consistent themes for every broadcaster at the IBC was the issues of logistics and NBC Olympics was no different. Throughout the games everyone’s operations were hampered by issues related to power, official bus transportation, traffic when not using official transportation, language barriers, and more.
“With the trucks coming in sort of late and with some of our shipments being delayed by customs we were pressed to get into the venues with the amount of time we would typically want which would be a couple of days,” says Chip Adams. “And this time, with all the construction going on and the where the trucks came in to port, it took them a day to get to the venues. So it was just a challenge getting everyone up to speed and moving everything in on a truly fast track install schedule.”
Chip Adams says that the NBC Olympics venue operations teams, across the board stepped up amidst some challenging conditions.
“We have good teams all around and they are attentive to what production wants to do,” says Adams. “Their main goal is making sure that the production teams get what they need to have the best production on screen.”
One production group that was impacted by the way the Rio Olympic venues were laid out was the diving team as the diving and swimming venues were in separate venues. That required two separate set ups and also deprived the diving production team of leveraging some of the equipment that has been on hand for swimming in recent Olympics. For example, in London they had access to two RF cameras, six edit suites between diving and swimming, and they also had five EVS servers. In Rio that EVS complement was cut back to two and the editing was actually done back at the IBC.
The fact that the opening ceremony and the athletics are not in the same stadium was another issue as going to either one during set up basically meant it was a full day trip.
The distance between the IBC and the studio at Copacabana Beach also caused some logistical headaches.
“There are certain support services you need to maintain a studio, and when the studio is here at the IBC there is space, access to writers, and a lot of support services that are a challenge to get on a beach that is being eaten by the ocean,” says Chip Adams.
Overall Adams says the venues themselves were in pretty good shape when the NBC Olympics arrived in force.
“What they weren’t ready for was some of the overlays we had, like tech, power, and data,” he adds. “So those three things were late and that was endemic across the entire 2016 Rio project. But facility wise we never had any complaints about being unable to get camera platforms in or having them built. All of those sort of things were ready for us. And the attitude of the Rio venue managers and the organizing team and the OBS venue team were very accommodating in helping us get what we needed done.”
Time for Reflection
The 2016 Summer Olympics saw the team from NBC tackle a number of new workflows and also deliver an unprecedented amount of content, both on TV and online. The team also, arguably, does more to add its own production touch and flair to its content than any other rights holder.
“We will look at the degree of difficulty and the complexity of things and that usually means figuring out how much risk something put us in,” he says. “Of course, just because something is hard doesn’t mean we won’t do it because that’s what we are paid to do. But sometimes it gets so complex we are on the edge of a cliff.”
Much of the risk and complexity in a project the size of the Olympics is because more and more of the technical and production infrastructure cannot be evaluated until it is in actual show mode.
“The time to configure the systems now exceeds the time it takes to wire it and that is very different from the past when it would take 60 to 90 days to wire everything,” says Mazza.
Terry Adams elaborates: “The physical stuff is not the hard stuff anymore. It’s watch folders, data paths, and naming conventions for all of this huge amount of stuff that is connected.”
That is one of the reasons it was so important this year for NBC Olympics to pre-configure the Rio system in Stamford, CT. Pre-configuration took place on the floor below the Highlights Factory.
“In Stamford we have the luxury of a maintenance staff and we can have a meeting in the morning and then go downstairs and work on configuring the system,” says Mazza. Previously it was a 90-minute trip to the Olympic field shop so a full-day trip is replaced with a quick trip downstairs.
“The whole thing is now file based so you have to have this great data structure in place,” says Terry Adams. “We were able to do that before we left for Rio and essentially all we did was make that fiber connection between upstairs and downstairs 4,000 miles longer.”
The continuing challenge for NBC Olympics and every other rights holder at the Olympics and even OBS itself is the continuing transition to IP-based workflows. A 17-day sporting event is an entirely different animal from any other sport production and a move to an all-IP infrastructure is still a few years off and requires a new way of thinking and problem solving.
“Trouble shooting in IP is an entirely different skillset and it’s going to take a while for the broadcast guys to move into an IP world,” says Terry Adams. “An IT switch taking 15 minutes to boot just is not acceptable.”
So there is plenty to consider with respect to the 2018 Winter Games, set to be held in PyeongChang, South Korea. The nature of the Olympics schedule means that there is a little more than 18 months between now and those games, creating an accelerated schedule versus the gap between a Winter Games and the following Summer Games. The Winter Games are also a bit smaller. And then, of course, there is the massive time difference as PyeongChang is 11 hours ahead of NBC’s Stamford facility.
“While South Korea does have a reputation for modern facilities in its cities that is not where we’re going as we will be in the middle of the mountains,” says Chip Adams. “So we will have to do a lot of due diligence with respect to power and data connectivity. Plus, we will need to be more buttoned up and do more of our integration prior to getting there as there are not a lot of Radio Shacks or cable vendors in the middle of the mountains.”
Adds Mazza: “It may not be visible at home how many of the things at the Olympics work, because it all is intended to look like any other high end truck based sports event coverage. A huge credit for that goes to OBS who does a tremendous job of absolutely first-rate host coverage. And for NBC, the complexity that goes into customizing the coverage of all 35 sports venues comes in two flavors: first, the logistics and local conditions of the larger venues for the four remote production units, four flypacks, and the huge IBC setup that we had here. And then second is the complexity of smaller venues using the mixed zones, commentary positions, remote controlled switchers and splits, 23 announce booths, 13 control rooms split across 4 cities, and all the circuit routing in both directions to allow the production to work seamlessly from two continents. Even for those of us who have grown into this over the years, it is still somewhat mind numbing in complexity. And add to that – that it all was working in real-time LIVE….suffice it to say…we are very relieved… and somewhat tired.”