Storage solutions for video archives: How long Is forever?
As the number of video-content platforms skyrockets, archive storage by sports organisations is expanding to new heights. Archive platforms continue to scale out, while the price per terabyte tumbles. Old- and new-media technologies (SSD, disk, tape) must be explored, for they all offer options for managing archive content. In the perpetual cycle of migrating content from ‘old’ to ‘new’ technology, the search for the Holy Grail is never-ending. At SVG USA’s recent 11th-annual Sports Content Management & Storage Forum, a panel chaired by Tab Butler, senior director, media management and post production, MLB Network, explored the current solutions for both large and small archives, with an eye on the future.
“When you’re talking about archives, the question you always ask [users] is How long do you want to keep that asset around and how fast will you need it? And you always get [the response] I want it forever and I want it instantly,” Butler said in opening the session. “So you have to ask How long is forever. Is it one cycle in technology? Two technology cycles? Is it when your boss is retiring? Or is it when the technology is retiring and you are going to have to transfer to another emerging technology. So there are a lot of challenges.”
A content tsunami: The more video you store, the more they produce
Sports-content owners like MLB Network and WWE have experienced an exponential explosion in content over the past decade. As a result, media managers have had to find creative new ways to expand their archives and intelligently store content.
In 2011, MLB Network produced 2,500 hours of recorded video content per week and had 245,000 hours of content in its archives. This year, the network is averaging 7,000 hours of recorded content per week and will have 885,000 hours in its archive by year-end.
WWE, which archives every single piece of content that comes in, has seen a similar explosion: from 650 hours of recorded content per week and 55,000 hours in its archive in 2011 to 1,280 hours per week and 200,000 hours overall, respectively, in 2017.
“Today, it’s so easy to acquire footage, and it’s so easy to store it that saying no is no longer an option,” said Bryan Staffaroni, senior director, media asset management, WWE. “We archive every single [asset] that comes in because the WWE production team never knows when they’re going to need something. We’ve run out of excuses as a service department to them for why we can’t keep everything. The beauty of the archive is, it’s limitless in storage space, so, if you fill up a robot, you eject your tapes and make room for more slots.”
For example, the WWE often ingests the feed from a 4X-slo-mo iso camera for an entire three-hour WWE event, which means that the MAM team ingests 12 hours’ worth of footage.
“Back in the day, when your library was getting full, you would be selective about the things we keep, but now we keep everything,” Staffaroni pointed out. “When [producers] go out and are acquiring footage, they shoot more because of the ease of it, and they know we [will store it].”
Managing ‘Data at Rest’ costs while maintaining fast access times
Butler identified the need to reduce cost for your “data at rest” as a primary future need in evolving MLB Network’s facility in Secaucus, NJ. Dell EMC CTO, Media and Entertainment, Thomas Burns noted that, although LTO storage helps to accomplish this, it does not necessarily make sense for nearline storage.
“Storage-at-rest costs need to drive down, and an LTO robot is a very efficient mechanical device. But, if you’re using it as an active archive, then you are weighting your access times in the wrong way,” he explained. “Tape is great if you just want to leave it there and do nothing with it.
“But I think archiving isn’t just about primary storage and deep archive anymore,” he continued. “There’s a whole spectrum that starts with very fast on-prem Flash storage for your production tier and then extends all the way down to something that you park in an archive because you don’t think it’s ever coming back. If you are using tape as an active archive, you need to look at another technology: like object [storage] either on-prem or cloud. That is the only thing that will allow you to replace tape with the concept of an active archive.”
Visibility is key: If users can’t find content, it’s worthless
As sports-video archives grow and producers demand more content at their fingertips faster, visibility is increasingly important. In building an archive ecosystem, media managers must provide quick access and easy search mechanisms for their production teams.
“One of the keys we’ve discovered is, when you’re working with archives — whether they are deep-cold or more-active archives — it’s important that you maintain a semblance of visibility to go along with your access,” said Dave Frederick, senior director, media and entertainment, Quantum. “If you don’t know what’s in there and regular users can’t find what they’re looking for, then that archive, no matter how cheap the media, becomes too costly to get things back out in terms of manpower and think-time. Having a system that maintains that visibility no matter how deep that archive is key to a strategy that allows you to leverage these low-cost formats.”
4K content vs. 4K storage: Separating church and state
The proliferation of 4K sports content is creating yet another challenge for media managers, one that will only grow in the coming years. As more and more HD content is delivered to the archive, the storage of 4K video will only compound the challenges. According to Erik Weaver, global director, M&E market development, HGST, the key to overcoming these challenges will change the mindset to separate the content from the storage medium.
“The problem you’re going to face from HD to UHD is an 8:1 ratio: you are going to face eight times the storage that you did with HD, which is going to start crushing you,” he said. “You have to start considering the concept of the separation between church and state. The idea is [that] what your content is and what your storage platform is are different. You have to figure out how to separate them, so that the platforms and mediums can continually evolve and change for you. One method to do that would be the AXF SMPTE standard. You need to be cognisant of both of those paths and where they are going separately.”
The lizard’s tail: Determining importance of data near-term vs. long-term
The emergence of advanced analytical tools has allowed sports-media organisations to automate much of the data-retrieval process. For Panasas, which has roots in the HPC (high-performance–computing) world, this has meant allowing sports-content owners to determine the value of their data depending on where a piece of media is in its lifecycle.
In determining the value of your data, said David Sallak, VP/CTO, M&E, Panasas, “think of it as a lizard’s tail: it’s long, and it gets narrower the longer it gets until it has a finite end point. For most of our media customers, a year of value is pretty long. Past that year, they will programmatically determine when they want that data back. There will not be the need for instant access; rather, it will be scheduled around an anniversary or another human-related event that drives the return of that data. That means, past a year, LTO and other [tape] formats are perfect. But the velocity and volume required for data to come back when it’s less than a year mean you are in the thicker part of the lizard’s tail where it is more likely to become important. That’s where we thrive.”