Live from the Rugby World Cup: Canon Free Viewpoint system ready to bring volumetric capture replays to semi-final matches
The Rugby World Cup is about to enter the crunch semi-final stage of the competition and the remaining matches at Yokohama Stadium will feature what has been the most exciting broadcast innovation at this year’s event: the Canon Free Viewpoint volumetric capture system.
It synthesises image data from 125 cameras located around the stadium in near real time and allows for a virtual camera operator to fly a camera through the action on the field and create 1080p replays from any angle.
The Canon Free Viewpoint system at the Rugby World Cup has 125 Canon cameras mounted around the stadium to capture matches from all angles.
Tsuyoshi Wakazono, manager, Canon SV Business Planning Department, Image Solutions Business Operations, and Atsushi Date, Canon’s senior principal architect, discussed the technology with SVG in Japan earlier in the tournament.
“We have received a lot of good feedback and it makes us very proud and happy that it is going well,” said Wakazono. To see the clips that have been created to date click here.
The Canon team has been hard at work managing the system that required 125 Canon 4K Cinema EOS cameras and lenses to be specially modified for the event. Those cameras operate in 60fps mode and have been mounted on 60 columns around Yokohama Stadium, a process that took approximately three days. The cameras have proprietary sensors that have been customised for the application and the data is then transferred to proprietary video servers in a small cabin located outside of the stadium. A team of 10 people are located in the cabin with five overseeing the computer control system.
System software extracts balls, players, and other objects of interest from the scene and 3D computer graphics processing to process those objects and players from multiple angles. A pilot then uses controls similar to those used for a cable camera system to virtually fly within the environment and then another team of five people oversee the image processing of the desired replay.
“We believe we can get the number of people who control the camera system down to two or three people,” said Date.
The process occurs in near real time, a major improvement over a system that only two years ago would take more than a day. The team is currently creating a minimum of six clips for each match and then making them available to rights holders who can then integrate them into their post-match analysis.
“We are working on a real time system and hope to make it available in the near future,” said Wakazono.
Depending on the sport being covered and the size of the venue less cameras can be used, a step that would increase the processing speed (likewise adding more cameras or higher-resolution cameras may improve the quality of the replay but the tradeoff would be longer processing time).
The use of the system at the Rugby World Cup is an exciting real-world application of a volumetric capture system that Canon hopes to commercialise in 2021. Permanent installations are one of the goals, but it has also been optimised for quick set-up for temporary events.
For example, the core equipment is located in two broadcast vans that can roll up to a stadium without impacting control room operations inside the venue. Date says the team also continues to improve the resolution, processing speed, and quality of the system.
Added Wakazono: “We are very happy with results but want to find a good partner in the future who sees the importance of real time instant replay and can help us reach that goal.”
Discussions are currently ongoing with potential customers of the technology and there are also potential applications beyond broadcast replays. Post-match analysis for coaches and players could take on a completely new level of usefulness and it also could be used to help with VAR and other video-assisted replay systems as it can offer a definitive look at whether or not a ball crossed a line.