View Full Version : Digital crisis: Motion pictures may fade to black


Tspeer
02-08-08, 09:31 AM
Fewer than half of all feature films made before 1950 have survived

http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9061099&pageNumber=1

Some interesting facts and comments from studios. Likely pertains to the future of HDM too.


February 8, 2008 (Computerworld) Current storage technologies may have a reputation for being plentiful and cheap, but not necessarily in Hollywood, where a recent study warns that the annual cost of archiving a digital film is 11 times that of storing celluloid film.

According to "The Digital Dilemma (http://www.oscars.org/council/digital_dilemma.html)," a report recently released by the Academy of Motion Picture Arts and Sciences (http://www.computerworld.com/action/inform.do?command=search&searchTerms=Academy+of+Motion+Picture+Arts+and+Sciences), digital film storage costs $12,510 per year, compared with $1,059 for celluloid. More dramatically, source materials -- those outtakes and audio recordings that often make up bonus content for special edition products -- cost 429 times as much to store, a whopping $208,500 per year for digital materials vs. $486 for film.

Image: The annual costs to store movies (http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9061579)

The annual costs to store movies
The report's authors state the data explosion could turn into digital movie extinction, unless the studios push the development of storage standards and data management practices that will guarantee long-term access of their content.

As the report points out, even if a 100-year black box were invented that "read data reliably without introducing any errors, required no maintenance and offered sufficient bit density at an affordable price," there would be nobody alive capable of repairing it if that box were to fail at 99 years. In the real world of data management, digital assets are stored on media with longevities much less than 100 years, vulnerable to temperature changes, humidity and static electricity. It can be misidentified, inadequately indexed and difficult to track.

Also, whereas a well-preserved 35mm negative has traditionally contained enough information to fulfill any requirement for ancillary markets, there's a question in the minds of some industry observers about whether the quality of masters archived in digital formats will be sufficient for quality duplication. In an age when home movie systems can often provide a better experience than some commercial theaters, that's not an unimportant concern.

"This is a clarion call to our people," says Milt Shefter, lead on the academy's Digital Motion Picture Archival Project and president of Miljoy Enterprises Inc. (http://www.miljoyent.com/), a media asset protection and preservation consultancy. "While there are great benefits to this technology, if you embrace it today, you are giving up guaranteed long-term access which you have with analog film."

According to Shefter, the price tag is of secondary concern to the studios. "The real cost is in not being able to guarantee that you'll have the material."


In digital we trust ...
When Paramount Pictures, under president Sherry Lansing, decided to create a 50th anniversary edition of Sunset Boulevard, one of the studio's numerous movie classics, it faced a dilemma. As described on the Paramount Film Preservation site (http://www.paramount.com/filmpreservation/projectssunset.html), the original movie was shot on cellulose nitrate film (http://www.filmpreservation.org/preservation/nitrate.html). The studio had been making prints off of reprinted movies for years.

The problem was, movie duplicates had deteriorated. The prints had become grainy, and contrast had built up to the point where the darker scenes were almost black and indiscernible. The copy that had survived in the best condition actually included "small but very visible digs" in every frame. On top of that, through 50 years of studio history, every original piece of the film footage by director Billy Wilder had gone missing, including the original negative of Sunset Boulevard.

So rather than releasing yet another degraded version of the film with added material that would allow the studio to call the resulting DVD a "special collector's edition," Paramount called on its archivists to remove the scratches and dirt and restore the movie to as close to its former glory as possible.

In 2002, the preservation team hired the services of Lowry Digital (http://www.dts.com/digitalcinema/)(now named DTS), which applied a proprietary process to the movie, in which all visible damage was removed from a digitally scanned version of the negative. That high-resolution digital file was scanned back to film to create a new negative and output to a master tape from which a DVD was created. The collector's edition (http://www.thedigitalbits.com/articles/robertharris/harris111502.html)of the classic was finally released six months later in that same year.

That potential for losing irreplaceable content is supposed to be reduced by digital moviemaking, right? Directors shoot their movies on digital cameras and perform post-production on computers; the studios distribute the films to theaters via hard drives, tape drives or satellite; and then cinemas show the films using digital projectors. The ones and zeros behind the process never degrade.


The -- gulp! -- cost of storing a movie
But those digital bits add up -- to 2PB of data for a single movie, according to the academy. When 35mm film is used, studios can easily save everything: the original negative, the original audio, the still photos taken on set, notated scripts and other material. One case study cited by the report states that a two-hour feature film would take up about 129 cartons or cans, which are normally stored in vaults often located in underground salt mines. "Nobody paid any attention to what the budget was because it wasn't significant," says Shefter. The report estimates an annual cost of $1,170 to store the original production footage of a particular movie.

Compare that to the expense associated with storing that production footage in a digital format on tape in a fully managed storage facility. The cost is about $1.6 million.

While a director using 35mm film might shoot 15 or 18 minutes of film for every minute used in the final movie, "that ratio goes up tremendously when you go to digital," says Shefter. "It encourages more use." For instance, because film doesn't need to be loaded into the camera, the cameras just keep shooting -- even as the director steps out from behind the camera to talk with the cast.

Adding to the amount of data created in the making of a typical movie are the files generated during the post-production process, when the footage is turned into a sellable product. Directors believe they have better control when the movie goes to digital. "You can do so much more in the post-production process in digital than [you] were ever able to do in film," says Shefter.

The bottom line is that movie studios are in a position of having to maintain hundreds of terabytes of data for the material associated with any single motion picture, content that's barely or rarely cataloged or indexed.


Choices in storage
According to Brian Babineau, an analyst at Enterprise Strategy Group Inc. (ESG), the two basic choices for digital movie archive media -- tape or disk -- differ in advantages and disadvantages. Tape, the less expensive media, is highly portable, but the backup is usually compressed in a proprietary format. Also, searching for data on tape involves looking for files within files -- akin to looking through Zip files. "Then if you want to read data back, you have to restore tape through backup software into its native format," says Babineau.

The pricier disk-based archives store the content in their native format. It's more flexible than tape, says Babineau. "With disk, you can search the index, and then through any software package typically recall it. If you need fast access, it's much more feasible to do with disk."



Image: Digital archiving forecast (http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9061238)
According to analysis by ESG, maximum tape and disk capacities are about the same in commercial and government sectors. This year, disk storage capacities are expected to pull ahead of tape. By 2012, the amount of data stored on external disk will surpass that of magnetic tape archive storage. According to ESG, disk storage will account for 34.3 exabytes of data in 2012 compared to 31.5 exabytes on tape storage media. Optical storage capacity, which is also tracked by ESG, makes a nominal showing, about 2.4 petabytes by 2012.


The 100-year goal
The significance of studio movie collections, which first surfaced with the showing of old movies during the introduction of TV in the 1950s, has only grown in value with the advent of videos, cable, DVDs and video-on-demand delivery services. According to data published by The Hollywood Reporter (http://www.hollywoodreporter.com/hr/content_display/news/e3ic5575a8c4f61aadd68a0d344f476d5da), only about 19% of total revenue for the six largest movie companies came from theater showings. The remaining 81% represents revenue generated through DVDs, TV, pay TV and VHS.

As a studio's revenue potential grows from post-theater releases of movies, the drive to digital platforms has sped up because of the need to be able to access content quickly. "We're seeing a lot of people pick and choose what types of content they keep on more accessible media, in order to generate incremental revenue," says Babineau.

Yet Hollywood's history of archiving has never been standardized. Many early titles, produced on flammable film, have simply been burned in warehouse fires or turned to vinegar in uncontrolled storage environments. Fewer than half of all feature films made before 1950 have survived.

Now acetate-based films and their related materials are more likely to be archived in climate-controlled facilities with fire suppression systems. The goal: a film master that lasts 100 years.

According to Shefter, digital tapes and disks that have replaced acid-free cartons and steel metal cans used for film "have not proved to be a significant successful method of preserving this information." Some users reported to the Academy of Motion Picture Arts and Sciences that the materials on the drives couldn't be accessed after only 18 months.

The question of the long-term reliability of disk storage was the topic of a study (http://www.eecs.harvard.edu/~mema/publications/eurosys-2006.pdf)performed by a group of university and vendor researchers in 2006. Their conclusion: While manufacturers of CD-Rs claim media "lifetimes up to 100 years ... actual lifetimes may be only two to five years." Media degradation happens not only as a result of "bit rot," a gradual accumulation of irrecoverable bit errors, making the data inaccessible, but also because media become obsolete. The scientists pointed to the once-ubiquitous floppy as proof.


The need for new processes
The major challenge that Hollywood faces in moving to digital has less to do with the actual limitations of the technology than with the reconfiguration of its processes, according to Steve Canepa, vice president of global media entertainment at IBM (http://www.ibm.com/sandbox/homepage/version-a/). "I talk about the factory notion," he says. "Today, you have literally dozens of ways that a piece of content can be viewed, depending upon the licensing rules, from theater to mobile to broadband to any number of devices in different formats. The methods and processes by which they go from higher resolution to lower resolution have to become increasingly more efficient."

That points to a major topic of the academy's report: the lack of standards. Studios are increasingly squeezing the window in which to monetize their movies, which means they frequently rely on archival technologies that are relatively immature. As media come and go, studios are forced to migrate to ever-newer platforms to be able to access their movie content. But that can leave content on older formats inaccessible. For example, LTO4, the current standard for tape drives in the movie business, which became available in 2007, is unable to read the contents of tapes written in the LTO1 format, the standard in 2000.

"We're seeing a lot of zero-day projects," says Babineau. "Whatever [companies] have right now in one format, they just keep it. But going forward they use a net-new type." The problem with that, he points out, is that the older content becomes unfindable unless equipment is kept around and maintained to read the older formats or there's a blending of indexes among formats.

"If you're dealing with a technology where you have to make a decision about what to do with it somewhere within a four- or five-year period, you have to know you're going to migrate it [or] get rid of it," explains Shefter. The studios' solution: to generate the majority of the revenue in that period before they have to make migration decisions.


Making way for what's to come
Migration projects shouldn't be a big deal to the movie industry, says Babineau. Having multiple copies of digital assets is simply a form of insurance against the loss of any particular master. "If you have something important -- The Godfather series, for example -- why not have a copy on film, but also digitize it and save a gold copy on disk somewhere? You can always go from disk back to film. We do it every day."

Canepa says IBM has customers in the film and broadcast industry that it helps migrate film to digital formats. Most of the time data is quickly migrated onto magnetic tape, which he says, is much denser and costs less. With compression, up to 1.6TB of data can be stored on a single tape cartridge today.

In fact, a similar migration project was announced in 2006 by IBM for Fox Broadcasting Co. -- not specifically for movies, but for its high-definition broadcasts such as NASCAR races, football games and shows such as American Idol. The broadcasts are beamed through satellites to an IBM xSeries server running Linux and then archived on magnetic tape.

Contrast that with ESPN, which, according to the academy's report, also runs a huge server farm. In the past, after a week's collection of broadcasts came in from professional and college sports, somebody -- usually an intern -- would go in and erase much of the data to make room for the next week's broadcast content. The process is described in the report as "triage on the fly." "That's a microcosm of what is going to happen in the industry," says Shefter.


The digital archivist
The job of the studio archivist -- the individual who, among other duties, decides what analog records need to be kept and maintained -- will become the responsibility of someone who is "more IT trained," predicts Shefter. "They will be in charge of distribution. They will decide what records they need to keep." One aspect of the archivist's work that will be lost in that transition, he says, is the ability to handle restoration. "It's sad, but I think that's the way it's heading."

Not everybody predicts the demise of future masterpieces with the advent of digital processes. In fact, the movement may portend even greater numbers of potential classics. Tom Streich, head of GripToyz (http://www.movingpics.net/griptoyz.html), a movie production support company in Utah, recalls when digital began to take hold 10 years ago. "It was like, 'Oh, everything is going to change. We won't need lights. We'll save so much money on not having to process film.'"

But, he says, the movie digitalization is not saving anybody money in production. What digital has done, Streich believes, is opened the industry to myriad new, low-budget film makers. "They can do it on their PCs and put out usable products for television. And once digital projection comes out big, then it's not bad for theaters either."

Doing the Math of Movies

120-minute film master 120-minute 4KB digital master

3 separations, cut negative, interpositive 120 minutes x 5 = 600 minutes of film; 3 copies of digital data files (4,096 pixels x 2,160 pixels x 6 bytes/pixel x 24 frames); 8.34TB x 3 = 25.02TB
Annual archival storage cost (43 cents/minute) $258/year; annual storage on data tape: $500/year/terabyte**

Production of archival masters $80,000

Production costs amortized over 100 years $800/year

Annual total $258 + $800 = $1,059*; annual total: 25.02 x $500 = $12,510


Source material for film production Source material for digital production

Shooting ratio 25:1 shooting ratio 25:1

120-minute movie 3,000 minutes of source material; 3,000 minutes of 4KB digital data; 208.5TB of data

Annual storage in "ambient" nonarchival conditions 16 cents/minute; annual storage on data tape: $500/year/terabyte
Number of copies stored: 2

Annual total $3,000 x 16 cents= $486**; annual total: $500 x 208.5 x 2= $208,500

* With rounding
** Estimate provided by San Diego Supercomputer Center

Source: The Science and Technology Council of the Academy of Motion Picture Arts and Sciences

WayneL
02-08-08, 09:44 AM
Good read

PaulGo
02-08-08, 09:55 AM
Thanks for the post!

oztech
02-08-08, 10:00 AM
same here thanks.

s2mikey
02-08-08, 10:17 AM
Hmmm..... Intesresting. Well, the studios and whomever else has a say in the matter ought to just release everything in High Def right away which will preserve the films for us. I mean...discs dont "fade" or "rot" away...right?

Wishful thinking, I know.... :D

Mr.D
02-08-08, 10:17 AM
there's a question in the minds of some industry observers about whether the quality of masters archived in digital formats will be sufficient for quality duplication

10bit or greater log encoded full negative density thats essentially transparent to the original 35mm?

Massive scaremongering by people who obviously haven't got a clue.

mhafner
02-08-08, 10:31 AM
That high-resolution digital file was scanned back to film to create a new negative and output to a master tape from which a DVD was created.

Stupid is as stupid does...

Mr.D
02-08-08, 10:42 AM
The pricier disk-based archives store the content in their native format. It's more flexible than tape, says Babineau. "With disk, you can search the index, and then through any software package typically recall it. If you need fast access, it's much more feasible to do with disk.

He doesn't know about the TOC (tabel of contents) that gets updated on the first file position of the tape then. Its an "index". And luckily enough if you inadvertently blow away the TOC ( if you overfill the tape) if you were smart enough to keep a copy of the TOC online (or even a print out) you can access the file positions directly by an incrimental read.

Tspeer
02-08-08, 10:45 AM
10bit or greater log encoded full negative density thats essentially transparent to the original 35mm?

Massive scaremongering by people who obviously haven't got a clue.

I took it as meaning that when such time comes around to rely on those digital masters (5 10, 20 years from when it was created) that the information could have degraded, or the storage medium has been replaced multiple times over during that time with newer technologies and the equipment required possibly no longer around.

However, I myself believe that now that they have recognized the problem, hopefully many older films are being remastered at least for archival purposes. This is a good thing. Also, I think technology is coming along in the next 10 years to allow digital storage on non-volatile ram which hopefully ends up being much more long term reliable that using tape or magnetic hardrives. Phase Change memory for example. A successor to flash ram. Solid state hardrives, or even perhaps a holographic storage for things like digital film masters should alleviate the current problems they've had where digital information can be lost.

Neo1965
02-08-08, 10:57 AM
I assume with older movies, the job of doing a proper digital remaster is likely the most expensive cost in the chain.

I wonder if there is a cheaper way to do this. If each frame can be scanned at extremely high resolution to say 4K x 3K onto YUV 4:4:4 at 12bits of accuracy, that's enough to just preserve their current state.

The subsequent color timing correction (with 12 bits to start with, there's a lot of freedom to go to 8bits) and more expensive expert handling can wait. If they just do this one step and keep the result on HDDs and do proper redundancy and regular refresh to move HDD over to new HDD, this content might last forever, or at least to the point when they can use automated tools that doesn't require the rare expensive human experts to step in and do their jobs.

Just an idea. Don't know if they can do that today.

This won't work if the restoration has to be done on damaged negatives though.

Icemage
02-08-08, 11:26 AM
I understand the point of the article, but does anyone else find it a bit strange that the article argues vociferously for continued use of film as the primary archival method, but starts off by talking about what a terrible shape Sunset Boulevard's film elements were in?

I don't doubt that it costs more for a digital master, but if film were so reliable, why do we have a foundation devoted to preserving films, and why does Lowry restoration even exist?

ewitte
02-08-08, 11:48 AM
Why pay so much for controlling the environment for digital video? I'd think it would be cheaper to just pay people every 5-10 years to copy to a new disc 20 copies each even.

Mr.D
02-08-08, 12:02 PM
I assume with older movies, the job of doing a proper digital remaster is likely the most expensive cost in the chain.

I wonder if there is a cheaper way to do this. If each frame can be scanned at extremely high resolution to say 4K x 3K onto YUV 4:4:4 at 12bits of accuracy, that's enough to just preserve their current state.

The subsequent color timing correction (with 12 bits to start with, there's a lot of freedom to go to 8bits) and more expensive expert handling can wait. If they just do this one step and keep the result on HDDs and do proper redundancy and regular refresh to move HDD over to new HDD, this content might last forever, or at least to the point when they can use automated tools that doesn't require the rare expensive human experts to step in and do their jobs.

.

Scanning is undertaken to at least 10bit log RGB using very well understood and established file formats. As long as the files are recoverable all someone needs to do is look up the specs of the file format and read them.

Unless civilisation gets destroyed at some point I doubt there will be any problems whatsoever retaining digital masters of films and if that happens the survivors will likely have more imporatnt things to do than watch films.

The article is scaremongering by observers who know little of the actual processes involved.

opfreak
02-08-08, 12:07 PM
I also find it amazing how the article brushes aside one of the major problems... That of recording everything. He mentions it, but then just moves on. I wonder what the total amount of 'film' there is today as oppsoded to back in the day. Its stated that on film the ratio was about 18 mins of film per 1 min of movie. Whats it today? 100:1?

eightninesuited
02-08-08, 12:10 PM
That was a very good read. I really enjoyed it. Didn't realize storing digital copies were so expensive.

Any ideas as to how long a Blu-ray disc lasts if it remains stored? Will it deteriorate over time? It think it's determined by the kind of dye formula they use in the disc, right?

I think studios will follow Warner and start making 4k masters of older movies. That's only good for us because it means more releases.

mproper
02-08-08, 12:17 PM
Any ideas as to how long a Blu-ray disc lasts if it remains stored? Will it deteriorate over time?

Yes.

On a related note, did anyone watch Life After People on the History Channel a week or two ago? The one lady brought up a good point about how modern man has no durable way to preserve things. If we all disappear, the books will rot, CDs/DVDs/Hard Drives will decay.

However, the cave drawings and Egyption carvings will far outlast all that stuff (although of course disappear eventually). Anyways, I thought it was an interesting point that we haven't come up with anything better than cave drawings to preserve a record of our society.

Ok, sorry to stray off topic...

RubberToe
02-08-08, 12:19 PM
You would think that people throwing around that kind of money would have a clue. Apparently not.

There are 2 basic fallacies that the people doing this are buying into. The first one is that you choose a "media" and then you will forever and ever use that media as your archive platform. Wrong...

Everyone in the IT business knows that every 5 years there is a brand spankin new archive media that is 1) somewhat more expensive than the previous media, 2) it will store 32x as much data as the previous media, and 3) that the new media will be completely obsolete in 5 years. Given this information, what you need to do as an archivist is pick a media that you will use for 5 years, and here is the important part, KNOWING that in 5 more years you will have to make another change, and migrate all your old media to the new media. They don't grasp the process as far as I can see.

The other problem with all the number crunching being done concerning costs is that they don't account for the fact that storage densities are exponentially increasing while costs per GB of data stored are decreasing at 30% a year compounded. That RAID array of disks as big a refrigerator today that you need to store this years movies will fit onto a holographic storage array in 5 years that is about 1/10 the size: http://www.dmxzone.com/ShowDetail.asp?NewsId=11267. And 5 years after that you migrate the whole shootin match to your brand new pocket sized nanoscale MEMS storage device: http://www.physorg.com/news3361.html. You get the idea...

While you can argue about which exact product or what exact density will be the "archive media of the moment" in 5 years, the one thing that is clear is that it will cost less than what it does today, and it will store 32x as much in a given area. There are thousands of people around the world doing research to make that happen, and the march of technology will only get faster, not slower.

I will store all the digital films they want for a fraction of the $208,500 per year quoted in the article, indefinitely, and provide them an exact copy of any film they want at any point in time for a nominal fee.

Sounds like I'm about to kick off a new business :)

RT

wakashizuma
02-08-08, 01:13 PM
Amazing read.

Figgie
02-08-08, 02:06 PM
He doesn't know about the TOC (tabel of contents) that gets updated on the first file position of the tape then. Its an "index". And luckily enough if you inadvertently blow away the TOC ( if you overfill the tape) if you were smart enough to keep a copy of the TOC online (or even a print out) you can access the file positions directly by an incrimental read.


The hard disk is still faster in access. Always has been. The downside has been the price compared to magnetic tape media.


Basically RubberToe hit the nail squarely on the head. 5 years tops for magnetic media. Then it gets replaced.

Neo1965
02-08-08, 03:35 PM
The hard disk is still faster in access. Always has been. The downside has been the price compared to magnetic tape media.


Basically RubberToe hit the nail squarely on the head. 5 years tops for magnetic media. Then it gets replaced.

Even with HDD, they can easily cycle digital data on HDD every 2.5 yrs and be very confident there is no generation loss. You can't go film to film to preserve it without some generation loss.

Otoh, I can see that the HDD can be placed on a shelf for ten years and not be expected to read correctly. The HDD after all like tape, still has magnetic properties too. However, some of those annual maintainance costs sound exorbitantly high. The HDD for example, 1TB is around $250 in retail, presumably that will be less in 2010. Presumably in 2015 we'd have higher density better archival storage, and so on, but the service charges seem incredibly high. As for that 80k production cost, that's what I was trying to figure out --- Are the film scanners really THAT expensive, or is that paying for the expertise to do it? That archival system is saying 6 bytes/pixel = 16bits for each Y,U,V, which is really high detail that might be an overkill.

But even then, HDDs don't cost that much, at consumer level, 12Mpel CMOS/CCD sensors are already reasonably priced. Someone could take what is in existing top end professional digital cameras, built some film advancing gear with a white light background, take each picture in RAW mode, save each frame sequentially, and that won't even take 10k in equipment costs.

To minimize needs for having to maintain/refresh the tape or HDD archive, for longer term storage, optical storage like BD-R (or DVD-R at lower density) for archival is supposed to last much longer than HDD, but I wonder if that's been time tested. Is there any gurantee that a BD-R (or DVD-R) can be shelved for 25 yrs and not have data errors when read?

Figgie
02-08-08, 04:27 PM
Even with HDD, they can easily cycle digital data on HDD every 2.5 yrs and be very confident there is no generation loss. You can't go film to film to preserve it without some generation loss.

Otoh, I can see that the HDD can be placed on a shelf for ten years and not be expected to read correctly. The HDD after all like tape, still has magnetic properties too. However, some of those annual maintainance costs sound exorbitantly high. The HDD for example, 1TB is around $250 in retail, presumably that will be less in 2010. Presumably in 2015 we'd have higher density better archival storage, and so on, but the service charges seem incredibly high. As for that 80k production cost, that's what I was trying to figure out --- Are the film scanners really THAT expensive, or is that paying for the expertise to do it? That archival system is saying 6 bytes/pixel = 16bits for each Y,U,V, which is really high detail that might be an overkill.

But even then, HDDs don't cost that much, at consumer level, 12Mpel CMOS/CCD sensors are already reasonably priced. Someone could take what is in existing top end professional digital cameras, built some film advancing gear with a white light background, take each picture in RAW mode, save each frame sequentially, and that won't even take 10k in equipment costs.

To minimize needs for having to maintain/refresh the tape or HDD archive, for longer term storage, optical storage like BD-R (or DVD-R at lower density) for archival is supposed to last much longer than HDD, but I wonder if that's been time tested. Is there any gurantee that a BD-R (or DVD-R) can be shelved for 25 yrs and not have data errors when read?

If "they" really wanted to. They can have a NAS, "Snapshot" the disc and make a backup of the snapshot to tape media and then take it off site. Which is typical for enterprise level environments. The problem is not storage but the power consumption of NAS/SAN.

The problem is that optical in general is the slowest of all storage medium. Not an issue if time is not essential but when backing up 10+ TB of data...... Then it becomes imperative that throughput be as high as the technology allows. Which makes optical a no go for massive amounts of data and hence why enterprise environments still use DLT/S-DLT/LTO etc.

Mr.D
02-08-08, 04:36 PM
Are the film scanners really THAT expensive, or is that paying for the expertise to do it? That archival system is saying 6 bytes/pixel = 16bits for each Y,U,V, which is really high detail that might be an overkill.



First scanner I used was about $3000000 I think. The laser recorder at the other end as it were was closer to 10 I think.

They are a bit cheaper now and faster per frame ( hence scanning costs have come down).

I'm not sure how much a Northlight is. I thnk its slightly under $1000000.

jpb123
02-08-08, 04:59 PM
And while we all think about these numbers maybe we should also think about if we should save everything for forever.

What will happen to humanity a thousand years from now if every painting, newspaper, book, movie, building has to be saved for forever. There will be no room for the people of that time to be themselves. Not when it comes to ideas and dreams and not when it comes to land to live on. Although movies may not be a factor there.

History has always meant that things and ideas get lost in time, not necessarily always a bad thing.

ewitte
02-08-08, 05:12 PM
Yes.

On a related note, did anyone watch Life After People on the History Channel a week or two ago? The one lady brought up a good point about how modern man has no durable way to preserve things. If we all disappear, the books will rot, CDs/DVDs/Hard Drives will decay.

However, the cave drawings and Egyption carvings will far outlast all that stuff (although of course disappear eventually). Anyways, I thought it was an interesting point that we haven't come up with anything better than cave drawings to preserve a record of our society.

Ok, sorry to stray off topic...

I saw part of it. No durable way? Nothing is keeping us from still putting something into stone... at least to get a message accross.

Figgie
02-08-08, 05:14 PM
And while we all think about these numbers maybe we should also think about if we should save everything for forever.

What will happen to humanity a thousand years from now if every painting, newspaper, book, movie, building has to be saved for forever. There will be no room for the people of that time to be themselves. Not when it comes to ideas and dreams and not when it comes to land to live on. Although movies may not be a factor there.

History has always meant that things and ideas get lost in time, not necessarily always a bad thing.

oohh how I wish this were true....

Library of Alexandria. That is what happens when "history" goes up in flames.

WayneL
02-08-08, 06:28 PM
I saw part of it. No durable way? Nothing is keeping us from still putting something into stone... at least to get a message accross.
Isn't the HDM disk stamper an example of a durable form of storage? It's not like electronic/magnetic storage, and shouldn't be susceptible to disk rot.

trygve
02-08-08, 08:21 PM
I also find it amazing how the article brushes aside one of the major problems... That of recording everything. He mentions it, but then just moves on. I wonder what the total amount of 'film' there is today as oppsoded to back in the day. Its stated that on film the ratio was about 18 mins of film per 1 min of movie. Whats it today? 100:1?

Heck, no. For an action movie shot on film you tend to figure on at least a 10:1 to 15:1 shooting ratio. Dramas can be closer to 6:1. I seriously doubt anyone serious is going for a 100:1 shooting ratio, not least because nobody wants to sift through all of that (or pay an editor and a team of assistants to).

Besides, the director, producer, DP, etc., would kill themselves after they'd tried to make it through the dailies for a couple of days and you'd have to get in a new production team twice a week as a result.

From a practical standpoint all that excess footage is not at all free. If it's on tape, that's a lot of handling, digitizing, logging, etc., you're paying for--plus you need to fill a datacenter with crap before you're ready to start editing. More and more, HD filmmaking is going to go straight to solid-state drives or other high-speed computer storage and those are not cheap to fill up like video tape.

Note, too, that even though your digital intermediary or other editing format may eat up a lot of storage space, your raw footage doesn't have to be quite so data intensive. For example, with the RED camera, one of your options is to record 24p/4K video in "Redcode RAW" format at only 27.5MBps--about 100GB per hour. That's very workable on even a modest editing system these days.

I think the price estimates they're using are scaled up with premium pricing all around. I have a few bookcases full of film reels just like everyone else and they don't cost hundreds of dollars a year to maintain--and I've had cut negatives and intermediaries stored at Fotokem's vaults for up to ten years free of charge...which they wouldn't do if it were costing them hundreds or thousands of dollars a year.

Sean_O
02-08-08, 08:29 PM
Those number$ are ridiculous.

lgans316
02-08-08, 09:53 PM
Very good article. Better to roll out these movies on HDM and prolong the digital age.

UxiSXRD
02-08-08, 11:19 PM
Moral of the story? Convert digital pics to celluloid for storage?

Kosty
02-09-08, 02:28 AM
Had a very interesting personal conversation with Joe Kane and a Hollywood film technician on this subject in Las Vegas last month at CES.

Heck of a thing that the best actual archive for even totally digital film is taking a master film copy an throwing it into a salt mine.

Some films have already had hard drive storage failures and some digital tapes have had data gaps that other backups have been needed. Joe Kane said for DVE, all the hard drives he had of it were frozen and unrecoverable an th D5 tapes (IIRC) he had multiple ones made all had bad data or unusable gaps ad that he was lucky that the bad footage or gaps were in different places and that he was able to use different tapes to reassemble the footage for his new version of Digital Video Essentials for HD DVD ad Blu-ray. He put a new master film copy in a salt mine I guess.

Plus the use of digital cameras an the tendency to let them run, instead of conserving film stock have made it more difficult to achieve the larger volume of not needed footage. Somebody has to make the decision now what digital extras are worth saving instead of just throwing all the extra film stock in a can and throwing it in a controlled storage environment. It actually costs more to storage digital stuff than film stock.

Moral of the story? Convert digital pics to celluloid for storage? That's exactly what Joe Kane and the film technician said. I guess thats what most studios are doing know, after the final digital edit, a first edition film master is made as a archival master for long term storage.

Hard Drives unless used arnot rliable over time as they wll freeze up. Optical media is untested over time and flash memory is too expensive and untested. Plus when stuff is on digital there ishust more of it than if it was on film and people conserve what is shot on film because of cost and digital is better to just keep running longer to capture more. Even the last final footage is expensive to back up in a digital way becuase its just now a lot and lot of bits.

Kosty
02-09-08, 02:48 AM
According to "The Digital Dilemma," a report recently released by the Academy of Motion Picture Arts and Sciences, digital film storage costs $12,510 per year, compared with $1,059 for celluloid. More dramatically, source materials -- those outtakes and audio recordings that often make up bonus content for special edition products -- cost 429 times as much to store, a whopping $208,500 per year for digital materials vs. $486 for film.

The final copy of the movie costs more because the film storage is simply putting into a secure climate controlled environment ( think salt mine) after a one time preparation. Digital is more because of multiple versions and taking it from the final digital work platform into a more secure digital storage form.

The reason all the work product (outtakes etc) source material costs more is that that there is jut more of it and someone has to review it and make a decision on what is worth saving , instead of just sticking all the extra stuff in a can and throwing it in a salt mine.

The biggest problem is not the hot blockbuster movies but all the not so successful long tail stuff. The best movies will be taken care of, the real issue will come in if it would take more to preserve a movie than people think it earned or will earn in the future.

Kosty
02-09-08, 02:56 AM
I understand the point of the article, but does anyone else find it a bit strange that the article argues vociferously for continued use of film as the primary archival method, but starts off by talking about what a terrible shape Sunset Boulevard's film elements were in?

I don't doubt that it costs more for a digital master, but if film were so reliable, why do we have a foundation devoted to preserving films, and why does Lowry restoration even exist?
Older film stock was more chemically reactive and was more carelessly stored. Studios just didn't it was as important or were surprised at its instability over long periods of time.

Newer film stock is much more stable over time, as is high density magnetic storage data tape, and is usually more carefully stored in multiple climate controlled facilities and more care is given to storing a early edition master copy that is not used for working replication usage.

Mr.D
02-09-08, 04:45 AM
. For example, with the RED camera, one of your options is to record 24p/4K video in "Redcode RAW" format at only 27.5MBps--about 100GB per hour. That's very workable on even a modest editing system these days.
r.

At the risk of being off topic I don't think the RED will break into high end film-making. The word I'm hearing from people who have put it through their paces ( and these are guys with years of working on oscar winning films) is that there are some fairly massive problems: compression artifacts, tearing on fast motion, inadequate exposure range.

Why do you think Jackson is now shooting Lovely Bones on film?

Mr.D
02-09-08, 04:49 AM
Joe Kane said for DVE, all the hard drives he had of it were frozen and unrecoverable an th D5 tapes (IIRC) he had multiple ones made all had bad data or unusable gaps ad that he was lucky that the bad footage or gaps were in different places and that he was able to use different tapes to reassemble the footage for his new version of Digital Video Essentials for HD DVD ad Blu-ray. He put a new master film copy in a salt mine I guess.



What the hell was he doing with it then?
Did he store it underwater ?
Or in direct sunlight on the roof rack of his car?

I've got beta sp anbd D1 tapes over 10 years old that can be read just fine and they've just been stuck on a shelf somewhere.

trbarry
02-09-08, 08:56 AM
...
Some films have already had hard drive storage failures and some digital tapes have had data gaps that other backups have been needed. Joe Kane said for DVE, all the hard drives he had of it were frozen and unrecoverable an th D5 tapes (IIRC) he had multiple ones made all had bad data or unusable gaps ad that he was lucky that the bad footage or gaps were in different places and that he was able to use different tapes to reassemble the footage for his new version of Digital Video Essentials for HD DVD ad Blu-ray. He put a new master film copy in a salt mine I guess.
...

That's an interesting point. It seems hard drives tend to fail all at once, maybe mechanically. For instance I've archived various HD movies to external hard drives and sometimes after letting them sit on a shelf for a year or two the lubricant turns to glue and they don't spin well.

But it sounds like on tape you instead get drop outs at random places. If this is true then it seems possible to design error correction such that if you had multiple copies of the tape it would be possible to combine them and have a good chance of putting it all back together even if all tapes had a few of the same blocks missing.

When a tape goes bad like that anybody know what percentage of the data blocks will usually be missing?

- Tom

trygve
02-09-08, 12:09 PM
What the hell was he doing with it then?
Did he store it underwater ?
Or in direct sunlight on the roof rack of his car?

I've got beta sp anbd D1 tapes over 10 years old that can be read just fine and they've just been stuck on a shelf somewhere.

Same here. I've yet to have one go bad. Recently had to digitize a few movies from 25+ year old one-inch type C reel-to-reel tape and those didn't have any problems either. I've had to go back to the original location sound tapes (10 year old DAT) sometimes and I haven't run into any unreadable ones doing that either.

pedrojunkie
02-09-08, 12:48 PM
You would think that people throwing around that kind of money would have a clue. Apparently not.

There are 2 basic fallacies that the people doing this are buying into. The first one is that you choose a "media" and then you will forever and ever use that media as your archive platform. Wrong...

Everyone in the IT business knows that every 5 years there is a brand spankin new archive media that is 1) somewhat more expensive than the previous media, 2) it will store 32x as much data as the previous media, and 3) that the new media will be completely obsolete in 5 years. Given this information, what you need to do as an archivist is pick a media that you will use for 5 years, and here is the important part, KNOWING that in 5 more years you will have to make another change, and migrate all your old media to the new media. They don't grasp the process as far as I can see.


Yeah but constantly transplanting terrabytes of data for each movie every 5 years for a catalog the size of a major studio like universal can't be cheap or easy. Plus you know that studios will stretch that out as long as they possibly can. I mean why the hell would they pay thousands of dollars every five years to maintain Howard the Duck? isn't it more cost efficient to just throw the masters in a salt mine and if you ever decide to release a dvd or a blu-ray or whatever in the future they can spend the money to refurb the master, it may be more expensive in the long run to refurb Howard the Duck but these guys have catalogs a million miles long, many may never see HDM or maybe they are going to be released at a distant time in the future where its cheaper to refurb than go through 3 generations of storage.

Finally hard drive failures are not an exact science, if a hard drive craps out without warning you will probably lose the film period. even with backups theres always a risk. Analog media degrades but it takes a long time or improper storage for it to become unusable. The studios should inspect the prints every 5-10 years or so just to catalog their condition and decide if it needs a refurb.

I guess the argument is that the studios are going to skimp on preventative maintenance unless it can be directly tied in with some kind of revenue its just the nature of business. Personally I'd go both routes since catalog releases bring in a butt ton of money on dvd and both storage techniques have a risk. And if you lose a film, its gone, but I don't know all the nuts and bolts.

I think most of the surprise/problems of the current state of affairs is that when it was stored no one really had a concept of how popular they would become. In the 50's it wasn't a dire need, in the 80's VHS and LD showed some interest in older movies, but later in the VHS world and then DVD took the world by storm, normal people started collecting movies old and new.

eightninesuited
02-09-08, 12:56 PM
See, I find this discovery scary, because when I read reviews of big blockbusters like Spiderman 1 being 'softer' than Spiderman 3 because it's not a new master, that says a lot about about the deteriorating process. Spiderman 1 is only 6 years old.

wakashizuma
02-09-08, 01:04 PM
Does HD filmmaking have a limitation when it comes to resolution as well?
I mean I think I've read somewhere that last 2 Star Wars were shot at 1080p; does that mean if we move to higher resolutions in future such as 4K, these movies have to be upscaled?
I used to think Digital filmmaking was the way of future but with this wonderful read I think I prefer film :D

By the way Kosty, How much space does a final master of a movie take? and at what resolution?

See, I find this discovery scary, because when I read reviews of big blockbusters like Spiderman 1 being 'softer' than Spiderman 3 because it's not a new master, that says a lot about about the deteriorating process. Spiderman 1 is only 6 years old.

I think Spider-man 1 could have looked better if Sony spent time and re-mastered the whole thing. I think they didn't have enough time or maybe didn't want to spend more money so they used the Superbit master and made a Blu-ray. If they can take old movies and make them look excellent (Close Encounter), Spider-man should be a piece of cake.

Figgie
02-09-08, 03:01 PM
Yeah but constantly transplanting terrabytes of data for each movie every 5 years .


5 years?

Not for anything but most fortune 500 companies send multi terrabyte information DAILY off site.

Hard drive for long term storage? no! NO! NO! NO! They definetly need a better IT management that is for sure.

Every company especially if they have to be SOX compliant must store backups for a 7 year time window. We always send year closing tapes off site and have never had an issue with data loss on over 500 TB woth of data. And that data encompassed CAD/CAM drawings, videos, research data and financials. So that leads me to believe that the "writer" in question did not use a properly temp AND humidity controlled environment with ESD precautions taken into consideration. Some thing like Iron Mountain offsite storage service for instance.

James R. Geib
02-09-08, 04:08 PM
I bet part of the costs have to do with hollywood unions and exclusive contracts. Surely there are less expensive and just as secure methods, but like the military, hollywood can only deal with 'certain' clients when it comes to ANYTHING related to a movie. A couple hundred lobster dinners are probably included in those costs! They shot themselves in the foot many years ago and the wound will never heal until there is some real change in the film industry.....

Figgie
02-09-08, 04:18 PM
I bet part of the costs have to do with hollywood unions and exclusive contracts. Surely there are less expensive and just as secure methods, but like the military, hollywood can only deal with 'certain' clients when it comes to ANYTHING related to a movie. A couple hundred lobster dinners are probably included in those costs! They shot themselves in the foot many years ago and the wound will never heal until there is some real change in the film industry.....


I would say it differently.

It is without a doubt NOT technology's fault but the blame rest squarely on the studio management decision (and speaking with 15+ years under my belt, studio seem to enjoy being as cheap as humanly possible on all thing IT).

BobRob
02-11-08, 12:56 PM
Hard drive for long term storage? no! NO! NO! NO! They definetly need a better IT management that is for sure.Agreed. Anyone archiving mission-critical data to HDD is a fool. Axiomatically, "it's not IF a hard drive will fail, it's WHEN."

But this, of course, to one degree or another, is true for all types of data storage.

And therein lies the problem. It's not just archival... it's archival and preservation. It sounds like the studios acknowledge the need for digitally archiving their assets, but are either unaware of, or unwilling to accept the fact that those archives need to be maintained. Also, technologies are forever changing. Data storage mediums are very here-today-gone-tomorrow. It's not enough to just digitally archive a film... that archive must be regularly tested for integrity, and migrated to newer storage mediums as required. It is not at all a passive process.

Icemage
02-11-08, 01:38 PM
Agreed. Anyone archiving mission-critical data to HDD is a fool. Axiomatically, "it's not IF a hard drive will fail, it's WHEN."

But this, of course, to one degree or another, is true for all types of data storage.

And therein lies the problem. It's not just archival... it's archival and preservation. It sounds like the studios acknowledge the need for digitally archiving their assets, but are either unaware of, or unwilling to accept the fact that those archives need to be maintained. Also, technologies are forever changing. Data storage mediums are very here-today-gone-tomorrow. It's not enough to just digitally archive a film... that archive must be regularly tested for integrity, and migrated to newer storage mediums as required. It is not at all a passive process.
There's another issue, though, and it's this:

There's no way to tell if a film element has deteriorated. Sure, sometimes extreme cases can be detected, but you've got no insurance.

With a digital copy, it is possible to tell when a digital copy has become corrupt. Keeping multiple copies so that damaged ones can be immediately repaired from working master copies is the key (that is, in fact, how modern systems like RAID arrays work).

If a scratch appears on a film master from physical trauma when moved or during the replication process, what do you do?

UxiSXRD
02-11-08, 02:05 PM
Agreed. Anyone archiving mission-critical data to HDD is a fool. Axiomatically, "it's not IF a hard drive will fail, it's WHEN."


A decent RAID cluster (RAID100 but certainly no less than 1+0) should be good enough, at least for short term shorage, with 100% monitoring by sys admins. This should be flexible enough to evolve but the key would be in making sure that the system adapted with the times and isn't maintained as is indefinitely (or until the supplies of spares is gone).

It'd be like finding a vault with the old punch card archives or tape reels that would be difficult. Periodic dumps to celluloid, if not Blu-ray in the meantime would be great. :D

Figgie
02-11-08, 03:12 PM
A decent RAID cluster (RAID100 but certainly no less than 1+0) should be good enough, at least for short term shorage, with 100% monitoring by sys admins. This should be flexible enough to evolve but the key would be in making sure that the system adapted with the times and isn't maintained as is indefinitely (or until the supplies of spares is gone).

It'd be like finding a vault with the old punch card archives or tape reels that would be difficult. Periodic dumps to celluloid, if not Blu-ray in the meantime would be great. :D


well here is the thing

Raid 1+0 is old tech. ADG (Advanced Data Guarding), Raid DP (Dual Parity), are the latest in RAID 6 technology. Except when the hard drives are powered off. ;) That is the issue these studios are running into. Archiving to HDD and then powering off HDD and sending somewhere for storage. WHich is the first red flag.

HDD archiving can be done but it needs to be done on a "Hot standby" model. Meaning the HDD in question need to physically be spinning and running. No data needs to get written to or read from after the archiving takes place but powered on is the key. Also the data CAN be accessed at any given time while on hot standby to check for data integrity but it is not the main useage of archiving on HDD.

UxiSXRD
02-11-08, 05:35 PM
Sure, the new RAID tech would be better and I definitely agree on the uptime/hotspare/monitoring (I had a customer once who declined to have me monitor his system and was "monitoring" his own RAID (but neglected his periodic checks and missed when it went into degraded mode. When he lost another drive, he lost the array and all his data).

I'd be more concerned about the server running an archaic OS/file system and there not being any easy way to get it off. With TCP/IP it shouldn't be an issue for at least the forseeable future, but I could see a future (say 100 years or so) where ipv4 isn't supported, the optical drives if any didn't work (if any burnable media can even be found), and some of these studios even if they did it "right" on the concept, hardware, and maintenance/monitoring, didn't have a way to easily get their data off without having to do some custom thing. And there's plenty of ways to screw up the rest, as well, mostly not having a significant budget (much less being stereotypically cheap).

Figgie
02-11-08, 06:14 PM
Well most SAN/NAS devices support

NFS (All *nix flavors)
CIFS (Windows FAT, FAT32, NTFS etc)
HFS (Apple)

so that covers just about all the OS that anyone would need.

<-- starts preparing my Resume to send to studio for storage expertise!! hehe