# Archiving in Raw vs. DNG



## icassell (Sep 6, 2009)

OK ...

I've been working in Bridge/PS for some time and finally broke down and bought LR2 a few months back.  Now I've finally decided to start using it and picked up a copy of Lightroom 2 by Nathaniel Coalson as an introduction.  He makes a very cogent argument for importing everything into LR2 as  Adobe DNG files and then dumping the RAW file.

I'm interested in knowing if there are good arguments AGAINST this practice.

Oh and let me apologize in advance if there are other similar threads.  I did a search and couldn't find one specifically on this question.


----------



## Derrel (Sep 6, 2009)

Well, the only person who I know has written AGAINST the practice of importing raw files into DNG is noted outdoor photography expert, digital technology writer, and Nikon expert Thom Hogan. He maintains that currently, there is still some room for improvement in raw image processing that DNG conversion removes. He also believes that it is not a good idea to have DNG become the defacto recording format for cameras, and that the manufacturers' raw formats can wring a bit more out of their sensors than DNG can.

I know that DNG conversions are roughly 20 percent smaller than many raw files in their native format, and with things like Fuji's huge 24.5 megabyte Extended Dynamic Range .RAF files, the conversion to DNG really cuts the size down....but it also removes the potential to use the BEST raw conversion software, which is Fuji's very own HyperUtilities suite,which still produces a better raw conversion than Lightroom,Aperture, ACR, SilkyPix, or RawTherapee.

Hogan believes that the DNG 'standard' is still not quite as advanced as it could be. Given how cheap storage is, I see no need to erase raw files,even if they have been cnverted to DNG files for smaller archives.


----------



## Garbz (Sep 7, 2009)

I see no reason to store two files if DNG has support for native embedding of the original data. Sure you end up with a 1.8x file size but you essentially get the best of both worlds. An open standard likely to be able to be read (see thread about guy who bought D300s and can't actually open his RAW file), and you get to keep the original data if in the future there are improvements to be gained.  Falls in nicely with the "given how cheap storage is" argument.

The only problem really with RAW that DNG is trying to address is lack of openness. NASA couldn't read it's own luna landing tapes. What choice do we have amongst a world of corporations who build obsolescence natively into every product. No one could with a straight face tell you that in 60 years, a propriety format which had to be reverse engineered at the time by everyone except the manufacturer has a higher chance of being read than a natively open documented format free to implement by anyone with time to write some code.

I see where Thom Hogan is coming from, but my counter argument is that this data is only relevant at the time of processing. As someone who processes an image once and then never touches it anymore for anything other than viewing or printing I actually don't see the point in keeping RAW files at all, when JPEG can display all the visible data in an sRGB file anyway.

So yeah the images may come out better if I reprocess them in 10 years, but they'll still look crap compared to my produces Nikon D5, and then really do I have the time to rework 250000 photos?


----------



## manaheim (Sep 7, 2009)

Wow, nice comments both Derel and Garbz.

I am someone who winds up going back to his images quite a bit.  Images I really like I wind up re-tweaking repeatedly over years, so I don't want to lose an fidelity at all so I've always been a bit leery of DNG.

I've been concerned about not being able to process those NEFs at some point, but I keep copies of the valid software that works (including very old operating systems going back to 1994 and even a few old PCs).  One way or another, I'll be able to convert them to whatever becomes the primary solution however many years from now... and I figure I'll likely see that change coming well before it happens anyway, so I'll do the conversion when I do.

This is just a summary of my approach to it, however... it means nothing more than this is what I have come up with that I am personally comfortable with.


----------



## icassell (Sep 7, 2009)

Hmmm .... clear as mud .... :lmao:

Thanks for comments, everyone.

I have to make a managerial decision here ... I don't think I'm going to convert my 50K RAW images to DNG, but going forward I have to make a decision ... there are good arguments on both sides of the fence ...


----------



## Garbz (Sep 8, 2009)

hehehe good luck. You got me thinking manaheim and that's always trouble. 

My first computer is only 15 years old, may still run if I can get the BIOS to recognise a replacement hard disk. I wonder if Windows 3.1 will run on modern hardware, certainly most of my old games don't run under Windows 7 (at all thanks to a drop of 8/16bit code support).

Kinda makes you want to grab a computer and store it in some sensitive archives somewhere, just in case.


----------



## Overread (Sep 8, 2009)

Garbz you might find this website helps feed your addiction:
GOG.com

though that said old games still can have problems - and display worse on newer computers than old - as software and hardware changes.


----------



## manaheim (Sep 8, 2009)

I ran Windows 3.1 on a Pentium Pro once. That's the last time I tried though.

And maaaaaaaaaaaaaaaaaaaaaaaaaaaaaaan did that sucker FLY. 

BTW, even when things don't work there are often emulators for older OSes, mainly because people love to play their games.


----------



## Garbz (Sep 9, 2009)

True. Lets hope emulators will run old photoshop. I see that as the only way to keep these archived files in the long run.

Adobe CameraRAW is now only a few years old and there's a HUGE list of files each proprietary and different from the next already: Adobe - Adobe Photoshop CS4: Digital camera raw file support If this keeps up then in 20 years the camera support component of the software alone will be many times the size of the actual program :S, something will give.


----------



## manaheim (Sep 9, 2009)

Garbz said:


> True. Lets hope emulators will run old photoshop. I see that as the only way to keep these archived files in the long run.
> 
> Adobe CameraRAW is now only a few years old and there's a HUGE list of files each proprietary and different from the next already: Adobe - Adobe Photoshop CS4: Digital camera raw file support If this keeps up then in 20 years the camera support component of the software alone will be many times the size of the actual program :S, something will give.


 
Yeah, true... though I wonder... I mean, at what point would you wind up being in a TRUE "oh crap I can't use this anymore" kind of situation?

In truth, you could probably download ACR and the cam definitions every couple years and just keep that on a hard drive somewhere... then you can always just go back and convert your old files if you hit that "Oh  $hit" point where suddenly Adobe drops support for your older cam.


----------



## Moglex (Sep 9, 2009)

manaheim said:


> Garbz said:
> 
> 
> > True. Lets hope emulators will run old photoshop. I see that as the only way to keep these archived files in the long run.
> ...



I would have thought that the number of times any normal person would *really* need to return to a RAW image and reprocess it would be pretty close to zero.

Saving in a TIFF with lossless compression would get you all you need in virtually every case (for most people, every case).

After all, with film you *never* got the chance to reprocess (although there were some rather gruesome 'rescue' processes you could apply to some negs), even one day after first processing.


----------



## manaheim (Sep 9, 2009)

Moglex said:


> I would have thought that the number of times any normal person would *really* need to return to a RAW image and reprocess it would be pretty close to zero.
> 
> Saving in a TIFF with lossless compression would get you all you need in virtually every case (for most people, every case).
> 
> After all, with film you *never* got the chance to reprocess (although there were some rather gruesome 'rescue' processes you could apply to some negs), even one day after first processing.


 
Define "normal person". 

As my skills have grown and my eye has adjusted over the years, I've gone back to many images and either drastically improved the final product, or found images of note that I simply didn't see before.

Plus it's just a matter of the whole "negatives" concept. Imagine if you couldn't pull your negative again (were you a film shooter) because some technology changed.

Unless I'm mistaken... the tiff doesn't have the broad exposure and color range that a RAW does...?  Right?  (Honestly, I've worked relatively little with tiff since I usually save in JPG for short term needs or PSD for long term ones)


----------



## Moglex (Sep 9, 2009)

manaheim said:


> Moglex said:
> 
> 
> > I would have thought that the number of times any normal person would *really* need to return to a RAW image and reprocess it would be pretty close to zero.
> ...



Fair point.



> As my skills have grown and my eye has adjusted over the years, I've gone back to many images and either drastically improved the final product, or found images of note that I simply didn't see before.



I can't argue with your experience but have there really been that many times that you've needed to go back to what is effectively the latent image and change the processing? If you had a correctly 'developed' image that hadn't been modified in any way would that not have been as good?



> Plus it's just a matter of the whole "negatives" concept. Imagine if you couldn't pull your negative again (were you a film shooter) because some technology changed.



But getting rid of the RAW (and keeping a processed but unmodified losslessly compressed image) is not getting rid of the negative, it's more akin to getting rid of the latent image but keeping the negative.



> Unless I'm mistaken... the tiff doesn't have the broad exposure and color range that a RAW does...?  Right?  (Honestly, I've worked relatively little with tiff since I usually save in JPG for short term needs or PSD for long term ones)



It's true that you would lose something moving from RAW to any lossless 'developed' format.

What I'm saying is that what you give up by moving to a non-proprietary format is only a very small amount of flexibility ( for TIFF, microscopic for DNG) but what you gain is the guaranteed ability to be able to get at your data.

It may well be that in 50 years time it's impossible to get at music stored in apple or sony proprietary formats (or indeed anything crippled with DRM) but you can pretty much guarantee that you'll be able to access MP3 or FLAC.

I think the options are:

1) Store in proprietary and risk not being able to process the image at all at some point in the future.

2) Store in non-proprietary and risk losing a tiny amount of re-processing flexibility

3) Store in both and accept the extra storage cost.


----------



## itznfb (Sep 9, 2009)

I've actually started storing files as jpg. I'm the type that doesn't like to delete anything.... even completely missed and out of focus shots. (I'm trying to break that habit). After I've processed the "keepers" I run ViewNX and convert all the raw files to jpg. If I need to go back and re-process something. Oh well, I'll have to make due with the jpg.


----------



## Garbz (Sep 9, 2009)

The other major assumption is that Adobe is still around in 50 years. They may get eaten up by Google, when one day Google consumes every technical thing on earth.


----------



## itznfb (Sep 9, 2009)

50 years is a long time. With technology evolving the way it is I'm not sure it's possible to plan something like this even 10 years down the road. As you said something like Google to put Adobe out of business in 5 years even. Highly unlikely but it is possible.


----------



## manaheim (Sep 9, 2009)

Moglex said:


> I can't argue with your experience but have there really been that many times that you've needed to go back to what is effectively the latent image and change the processing? If you had a correctly 'developed' image that hadn't been modified in any way would that not have been as good?


 
Does it happen often?  I would say it happens occasionally... not often.  It happened this week... I noticed I oversharpened an image... a good example of something you cannot recover from with a TIFF.



Moglex said:


> But getting rid of the RAW (and keeping a processed but unmodified losslessly compressed image) is not getting rid of the negative, it's more akin to getting rid of the latent image but keeping the negative.


 
errr... I don't see how you would say that.  Maybe I misunderstand what you mean by latent image.

The negative is that which captures the broadest range of available image data for processing... it is the least developed part of the imaging process (shy of an undeveloped negative, of course)

The RAW file is far more analagous to a negative file than it is to a processed TIFF.  I understand you have more variance with a TIFF than you would, say, a print... but the RAW file has the broadest range of available image data for processing, and is the least developed part of the imaging process.



Moglex said:


> It's true that you would lose something moving from RAW to any lossless 'developed' format.
> 
> What I'm saying is that what you give up by moving to a non-proprietary format is only a very small amount of flexibility ( for TIFF, microscopic for DNG) but what you gain is the guaranteed ability to be able to get at your data.
> 
> It may well be that in 50 years time it's impossible to get at music stored in apple or sony proprietary formats (or indeed anything crippled with DRM) but you can pretty much guarantee that you'll be able to access MP3 or FLAC.


 
I've been in technology for eons now... erm... god... almost 30 years.  I've seen formats, operating systems, and whole computer technologies come and go... and to date I can still recover past technological lives with almost zero effort.

What's more is I don't think I've seen a single format die so suddenly or so quickly such that you would not be able to either convert your files in the waning light of the format... and what's more is that I cannot think of a single time when a format _actually_ evaporated, even when for all intents and purposes it actually should have.  GIF and the "ARC" (pkxarc) compression format being excellent (though very dissimilar) examples.

So, basically... on the REALLY unlikely chance that .NEF (or select versions of NEF, or some other brand equivelent RAW) start to be phased out... 

1. We will all know well in advance.
2. We will have tools available for many years that will still be able to read the format.
3. We will have tools available for many years that will still be able to convert the format.
4. On the off chance that we crawl under a rock to emerge 10 years too late, we'll still have older tools on CDs lying under 5" of dust that will still be perfectly valid for converting the old format.

So, really, what I'm saying is if you are converting to a different format out of fear of not being able to pull up your RAW images again at a later date... well, I think it's a bit more panicky than warranted.

What's more is that if you choose to convert to DNG, I'd say you're stepping out of the frying pan into the fire.  After all, it's just another RAW format... and whose to say that one won't die?  In fact, it hasn't been fully adopted by the vendors so it probably has just as much chance of success as Nikon's .NEF does of becoming the de-facto standard for Canon. 

I kid, but I assume you see my point.



Moglex said:


> I think the options are:
> 
> 1) Store in proprietary and risk not being able to process the image at all at some point in the future.
> 2) Store in non-proprietary and risk losing a tiny amount of re-processing flexibility
> 3) Store in both and accept the extra storage cost.


 
I think the possible risk in 1 is very very low.  I might be bold enough to say the risk is very very low as a matter of fact, not opinion.

I think the risk in 2 here is kinda grossly misrepresented.  You assess it as tiny, but it is variable based upon what you do to your image, how skilled you are, how much your opinions change on interpretations over time, etc.  I suppose you could keep multiple COPIES of your TIFF as you make changes which would SOMEWHAT reduce this problem... but that seems a bit crazy when you can just keep the RAW.

Three seems a good compromise if you're really that concerned, but again... I see it as way too much work and effort given the true risks involved.


----------



## Moglex (Sep 10, 2009)

Having given this a bit more thought I think you are correct on saying that RAW is the better way to go.

If you're really thinking about archiving something for 100 years that may be a different matter but for those of us who are mere mortals what you're really balancing is an immediate reduction in flexibility against the possibility that some images will become unreadable many years in the future.

Presumably people will have their best images stored in TIFF or very high quality JPG anyway so you are really only worrying about someone deciding 10-15 years down the road that they must retrieve an image they have never before thought worth processing.


----------



## Moglex (Sep 10, 2009)

manaheim said:


> Moglex said:
> 
> 
> > But getting rid of the RAW (and keeping a processed but unmodified losslessly compressed image) is not getting rid of the negative, it's more akin to getting rid of the latent image but keeping the negative.
> ...



The latent image is the undeveloped image.

It depends on exactly how you look at the 'developing' analogy.

The physical developing process can be considered an analogy to RAW processing because it is possible to push or pull the development and use a more or less contrasty developer.

So in that sense the RAW image is akin to the latent image.

What you get with digital that you don't get with film is the ability to 'have another go' at the 'development' if you don't like your initial (or an subsequent) result.

So the analogous elements are:

RAW image == Latent image
Processed RAW == Negative
Post processed RAW* == Print

* And the print itself, of course.

Of course, if you wish you can consider the RAW to be the equivalent to the negative but that doesn't work for me because you can do thing to the RAW that you can't do to a negative (e.g. push process after you've already developed).


----------



## manaheim (Sep 10, 2009)

Moglex said:


> The latent image is the undeveloped image.
> 
> It depends on exactly how you look at the 'developing' analogy.
> 
> ...


 
Ah ok, I get your meaning.  I wasn't aware you could have variable results in the process of developing the negative, though it instantly makes sense the second you mention it.

Well, I guess then the best way to say it is "a raw image is as close to a negative as you'll ever get".


----------



## Moglex (Sep 10, 2009)

manaheim said:


> Well, I guess then the best way to say it is "a raw image is as close to a negative as you'll ever get".



True, and in the sense of processing flexibility, at least, it is superior to a negative.


----------



## manaheim (Sep 10, 2009)

Awesome.

Hey, nice chat.  Thanks, I enjoyed it.


----------

