Thursday, 23 August 2012

HD Video Material - Long Term Storage

The more I think about the long-term storage of video material the more confused I get and the more difficult it seems to be to preserve the very high-quality original files that are generated by recording in 1920x1080p resolution. The problem to me seems to be that the original files aren't really that large but you can't satisfactorily edit with them and the working files that you have to generate in order to edit the material properly are extremely large and essentially impossible to store for the amateur user. I overcomplicate things even more for myself by having this desire to always to be able to go back to the original material if I want to re-edit it for some reason and this gives me added storage complications.

For some while now in my HD video camera I have been using 16 GB capacity cards and this allows me to store well over two hours of material on a single card, this is actually far more than I ever practically need for a single project.

Given that any standard DVD will hold over 4 GB of material I am seriously considering switching to using 4 GB memory cards, each of which will hold about 25 minutes worth of original compressed HD video. My theory is that if I go down this route I can simply keep all the original files from a particular project on a DVD, I can then generate the working files from that and do whatever editing I want to do, and finally I can delete the working files and still retain my original reference DVD and recreate the working files from that in the future should I need to do so.

This may sound like a long winded approach but at least it will always allow me to go back to the original material and will satisfy my desire to not throw anything away!


  1. Can't you just dump the data to HDD after using the 4gb or 16gb card? HDD storage is cheap so you can use multiple drives for backups, maybe a NAS for automatic backups and then upload stuff overnight for offsite backups. Dropbox, AWS Glacier, erm..

  2. Yes, that is definitely an option and I've been doing that up to this point. The problem I'm finding is that a drive fills up fairly quickly and also presents a "security" issue in my own mind. Pound for pound DVD storage comes in at about 7.5p per GB and HD storage comes in at about 10p per GB, depending on what you buy of course, so there isn't much in that.

    My concern is that if a HD goes up in smoke/gets dropped/put near a magnetic field etc. etc. there is the potential to lose original material from many many projects which would be very bad news. The chances on the DVD front is that there might be the odd read failure (reduced if one verifies the disc after writing it) but that would only involve one project.

    I know exactly what you are saying and it's very difficult as there are pros and cons but if I stick everything onto HDs backing them up then becomes another issue and historically, probably because I'm old-fashioned, I've found it easier to keep track of stuff on DVDs!

    It may of course all go full circle and in a few months you'll see me posting that I've changed my mind, again!

  3. I forwarded your question to my cousin who has recently completed a post-graduate course in Digital Curation. Here is her "text book" answer:

    Yes, HD does have the problems outlined. However, assuming he uses a computer to edit (and does a 'back-up' of his computer system) this is a good back-up option. (By-the-way deleting stuff, and 'recovering' from your back up method isn't 100% reliable as back-ups have been known to fail.)

    Standard DVCs have a 5-10 year failure rate (and can go before then). It would be a bit dodgy assuming equipment 10 years down the track will still read the DVC disks. Gold DVD do seem to last longer (I've previously looked into this) however heat (and UV light depending on what 'writing' method is used) will considerably shorten the lifespan of any DVD.

    Memory sticks (and I assume memory cards are similar) have a reputation for sudden loss, i.e. they just stop working. I wouldn't solely reply on them.

    Sudden, catastrophic failure is an issue in permanent archival digital repositories. It's addressed by using check-sum programmes looking for any changes/i.e. potential corruption before the technology can not longer compensate. The volume and type of data errors can be reasons for 'sudden' failure.

    The other method used by digital repositories is ...wait for it...keeping lots of copies (seriously!), and it's actually called the 'Keep lots of copies' method. In digital repositories this usually involves creating several different types of files, e.g. word, and .pdf and .rft etc. This method assumes records will from time to time be found to be non-recoverable, and a new copy is made from a duplicate copy to replace the failed copy. If done properly many centuries should be possible.

    Archivist keep records for really long periods of time. If a disaster can happen, given enough time it probably will (more often than most people realise), e.g. fire, floods, water, UV light, pests, war, tsunami, earthquake, theft, destruction and media degradation (everything breaks down over long enough times), etc. How high is his tolerance for loss? This will influence what methods of 'back-up' and risk mitigation are most appealing.

    Suggested options

    1) Keep the original HD cards, he can reduce risk by using smaller capacity cards.
    I think deleting the working files sounds reasonable, as long as you check the original material still works. Keeping one project on one piece of media (assuming good indexing and storage) also sounds good, and if one piece of media fails the others might still be OK.

    2) Use Gold DVD as a back-up method (maybe for several projects).

    3) Investigate whether he can transfer the material onto this computer (without creating large files), back this up as per PC.

    Risk reduction methods

    1) Keep it away from heat (nothing over 25 degrees C which is very damaging), and sunlight (can fade inks), as well has the usual magnetic fields, and dropping.

    2) Don't store everything together.

    3) Check DVD's are working at least every 5 years and consider migrating them to a new DVD.
    However the more times they're migrated the higher the error rate (although this might take 20-100 years before being an issue). This could be mitigated by running a check-sum before and after copying.

    4) Don't let formats get 2-3 versions behind before migrating them, run check-sums to look for any changes after copying.

    5) Keeping 'files' in the live system is considered the most reliable method at the moment.

  4. Extremely helpful reply, very much appreciated and all the points are of course extremely valid. I guess my feeling is that my material is never really going to be "critical" though to me of course it does have value, at least for a while until I'm too old to be bothered!

    A further point is that samples of most of my material are available online on YouTube/online web galleries and so in theory at least that provides a further layer of security for at least some of the material which I hope is backed up on the Google mega-servers!

    There are also of course cost factors, keeping the original memory cards is too expensive an option for me and also the use of the gold covered DVDs is fine apart from the cost, which is well over 5x the cost of standard DVD-Rs and CD-Rs. I'll reserve the gold discs for anything really important!

    All in all much food for thought but I guess that I'm going to end up ploughing on with my DVD plan and keeping them in a dark room in a cool metal box. However it ends up I'm going to be better off than I am now! I'll be sticking with my one DVD per project from here on and that should at least reduce my risks, given that material that's important to me is available online too.

    All that said, disaster could still strike, but it wouldn't be life-threatening even if I lost the lot!