Using DNG safely

I will not be liable for any problems arising from the use of the information on this page. If you act upon the information on this page, you do so with the understanding that you agree to hold me harmless against any losses, damages, and costs arising from your action.


What are the issues?

Photographers want to know whether using DNG will screw up their photography, or conversely make it safer, now or in the future!

What is the alternative?

It isn't sufficient to judge a DNG-based workflow in isolation. It is also necessary to compare it with any alternatives. DNG rarely replaces a perfect workflow. It typically changes a workflow that already has points of failure. Some of those points of failure may only matter in years to come.

Some of this page is about ensuring that your raw image files, DNG or not, are resilient to any "single point of failure", such as loss of a disc or even loss of a whole computer. If your workflow is sufficiently resilient, perhaps DNG will make little difference to its resilience. If using DNG at all is seen to be a safety problem, it is worth checking whether there is a problem even without DNG!

For example, some people complain that converting to DNG means they can't use products that can't handle DNG. Does that mean that they have discarded their native raw files? So are they comparing "having just one copy of their native raw files" with "having just one copy of the DNGs"? Yeuk! One copy is very risky! If they didn't use DNG, they would still be safer having two copies of their native raw files. And if they did that, why not replace one of those copies with DNG?

Topics

  1. If you convert to DNG and discard your original raw files, and then need to use software that doesn't support DNG, then of course you are screwed. Much of this software is listed at: Products without explicit DNG support.
  2. You would also have problems if some of the data in your original raw files wasn't transferred to DNG, and that extra data ever became important. Do DNG files contain all the original raw image data?
  3. Conversely, the conversion to DNG may provide an early-warning of problems before it is too late.
  4. Photographers who decide to use a DNG-based workflow want to know how to do so safely. What are some important considerations for workflows?

This topic needs sober analysis, not "fear, uncertainty, doubt". This page is a contribution to that analysis. It discusses whether you can safely use DNG in your workflow. It also illustrates methods for making your use of DNG, and indeed use of other raw formats, more resilient to failure.

Competitions needing proof

Some photographic competitions that demand to see the original raw files of the winners. There are anecdotes that some of them may not accept DNGs from those raw files, even though they hold the original raw data.


Do DNG files contain all the original raw image data?

Detailed technical documentation about Adobe's support of DNG tends to be fragmented. Detailed information about that support's capability or about potential problems may appear in forums, and eventually disappear when the forums are tidied. This table attempts to capture such information until it is published at a better location. It also contains some information about the quality of the information within DNG files. Some duplicate and historical information has been moved to another page:
Reports about completeness & bugs
.

For those concerned about loss of information, there is a discussion below about getting the best of both worlds:
What are some important considerations for workflows?

Topic Links Commentary
Conversion other than Sigma / Foveon cameras

DNG Workflow Article

PhotoshopNews
2005-05-23

Support of black masked pixels in 3.2 which was released 2005-09-27

Jeff Schewe: "DNG Converter 3.1 and above does indeed move all EXIF private maker note from all TIFF-EP based raw file formats, not just NEF and CR2 files. The file formats that do NOT use EXIF based private maker notes are not migrated. File formats from; Kodak, Foveon and Leaf do not use EXIF private maker notes, so those files are not supported for migration. Additionally, Canon CRW files and Fuji RAF do not completely adhere to the TIFF-EP and EXIF spec. So some metadata will be migrated and some will be stripped....
" So, if you shoot NEF or CR2 you will, in essence, loose nothing by converting to DNG. Even the Nikon D2X encrypted As Shot white balance is moved to the converted DNG file. It isn't decrypted, but it is safely moved. As a result, if, at some time in the future Nikon grants Adobe permission to decrypt the WB data, it'll still be there in the DNG file waiting to be used, as will all the other undocumented metadata.
" If your Raw file format is not NEF or CR2, some or most of the proprietary, undocumented metadata found in the private maker notes of the EXIF specification will be stripped out. Is this important data? Hard to say since it's undocumented. However, it's metadata that Camera Raw can not currently use so if you are satisfied with the way Camera Raw can convert your file, then the proprietary metadata may be meaningless to you".

GroupBrowser
2005-05-23
Thomas Knoll: "The code change in ACR/DNG 3.1 that preserves private MakerNotes applies to all formats that store their private data using the Exif MakerNote format, which includes Pentax. These MakerNotes are often byte order or offset dependent, so what I'm doing to preserve this data safely is repackaging it into the DNGPrivateData tag. This repackaging stores the source file's byte order and the original file offset to the MakerNote, along with the MakerNote data itself. This allows an application that understands the MakerNote format for a particular camera to find the MakerNote data, along with the original byte order and file offset, and thus parse the MakerNote data correctly from the DNG file".
exifprobe - camera image file probe This discusses whether raw formats are TIFF-based or not. It is not clear whether the "TIFF & TIFF- derived "raw"" formats satisfy the MakerNote principles above.
DNG Workflow Part II

GroupBrowser
2005-05-25

PhotoshopNews
2005-05-25

Jeff Schewe: "Another little known benefit of using the Adobe DNG Converter is the ability to do a simple verification that the raw file you are converting is not corrupted. If the DNG Converter encounters a file that has problems, it will throw an alert error in the Conversion Status dialog as shown above.
"While rare, it's an early warning sign that there may be problems with either that file or the directory structure of your compact flash card. Generally a simple finder copy will not show an error or problem when copying a corrupted image file-it merely copies the corrupted file and you won't discover the problem until later when trying to open the file. Note that this warning is not a complete fail safe mechanism. It's merely intended to alert you to a potential problem and indicates which file or files are involved. This will give you a warning that perhaps stronger recovery may be needed before removing the card and reformatting it".

ORF conversion to DNG - good or bad results?

ORF vs DNG

DPReview
2006-05-26

FourThirdsPhoto Forum 2007-02-27

DPReview
2007-02-27

Michael Meissner: "Olympus uses a screwy encoding of the MakerNotes, and if you just rewrite the MakerNotes, you will write incorrect information (in some cases Olympus puts the wrong length information, in other places, it uses absolute addresses within the file, which have to be adjusted whenever the file is rewritten)."
Boardhead (Phil Harvey, ExifTool author): "The Olympus maker notes contain pointers which reference information outside the makernotes data block. This information (a few kB worth) is not copied by the DNG converter (I just tested DNG Converter version 3.4 and the problem still exists). To properly copy ORF maker notes, you need to understand the structure".
marlof: "Then I run a batch file converting the ORFs to DNG files, and using ExifTool to copy the EXIF info from the ORFs to the DNG files". (Documented here: DPReview).
Douglas Palmer: Windows & Mac scripts here.
DPReview
2007-04-27
Michael Meissner: "Given that the E-410, E-510, and presumably the P-1 use a new compressed RAW image that first showed up with the SP-550, you might have to wait a bit for the ORF->DNG converter to catch up. As far as I understand it, Olympus also fixed the bogus way they stored the MakerNotes in the new RAW file, so you won't have to depend on something like exiftool to restore the MakerNotes if you use that information".
"Canon 1D Mark III difference" DPReview
2007-06-11
Barry Pearson: using the DNG route "A small number of pixels differ by 1 in one of the channels".
"PEF differences"

DPReview
2009-09-03

Adobe DNG Forum
2009-11-22

GordonBGood: "The program will not work with DNG files converted from PEF's by the current version of the Adobe DNG Converter application due to this program stripping out the necessary black masked-to-light photosites at the right and bottom borders of the sensor in landscape orientation which are used by the correction algorithm".
Repeated in an Adobe Forum.
"My Adobe DNG chat with Eric Chan" Nat Coalson
2011-11-29

Nat Coalson: "When comparing the pure, raw, unmodified mosaic data in the original raw capture, is there any difference at all between that data and the image data contained in the DNG?"
Eric Chan: "The shortest and most honest answer is: it depends!"
(Followed by details).

Sigma / Foveon X3F conversion
Conversion of Foveon X3F files to DNG

DPReview
2007-03-26

DPReview
2007-03-26

Zalman Stern: "X3F files are similar to DNG in that the format is designed to contain all the info that a converter needs to turn the raw file into an image. Thus the Foveon provided code used in prior versions of ACR converts the SD14 files as it did with SD9/SD10 files.... The issue is that Foveon has evolved their algorithms for doing the conversion and thus can offer higher quality with their new code. This code is not in ACR 3. or 4.0".
Zalman Stern: "There is no way to support Foveon X3 data in DNG without publishing intimate details of the off-sensor data, which is Foveon's intellectual property".

What about bugs and deficiencies in key software products?

Bugs and deficiencies in DNG-handling software do occur, as illustrated in the table below. Sometimes the problem is actually with ACR rather than the DNG Converter, or even with the camera manufacturer. Typically the work-around is to not take the upgrade concerned. Typically the bug is solved in the next upgrade. Another problem is with unofficial support of new cameras, which is sometimes not up the standard of official support. Once again, typically the issue is solved in the next upgrade. Some duplicate and historical information has been moved to another page:
Reports about completeness & bugs
.

To avoid these problems, I test each upgrade of ACR and the DNG Converter before adopting them:
What are useful tests before adopting any upgrade to ACR or the DNG Converter?

Topic Links Commentary
Problems with version 3.6 & 3.7 (Mac)
DNG vs NEF White Balance Mismatch GroupBrowser
2007-02-21
Cajun Man: "D2X ... I have found that my photos often have very different White Balance values -- for both Color Temperature as well as Tint -- when comparing the original NEF and a DNG created by Adobe DNG Converter".
Thomas Knoll: "There is a bug in the Macintosh version of the DNG Conveter 3.6 and 3.7 in the handling of the white balance from these Nikon cameras. Either go back to version 3.5, or wait for 4.0. Or save the DNG files directly from the Camera Raw plug-in.... If you are brave, you can try to fix this yourself. Open the package (control click and pick "Show Package Contents") for the both DNG Converter 3.7 application and the Camera Raw 3.7 plug-in. Navagate inside each package into the Contents/MacOS folder. Copy the file "NkMiniLib.dylib" from the Camera Raw package into the DNG Converter package".
Problems with version 4.5
DNG converter 4.5 command line broken GroupBrowser
2008-08-11
DNG converter 4.5 needs full path names in its command line.
Example: "Adobe DNG Converter.exe" -c d:\pictures\PICT0001.MRW
The older versions did not require the full path name and would also work like this:
"Adobe DNG Converter.exe" -c PICT0001.MRW provided that the working directory was "d:\pictures".
Problems with Lightroom 1.4 (Windows)
DNG Conversion Error(Windows Only) Lightroom Journal
2008-03-16
"Adobe has included technology to verify that the image data in a DNG file is unchanged from when it was originally converted to DNG. ... when converting to DNG using Lightroom 1.4 on Windows, the application will write an incorrect verification tag to describe the image data. When Lightroom attempts to work with those files in the Develop module, the application reads that incorrect tag, believes that there is something wrong with the raw data and will present an error. ... there is nothing wrong with the integrity of your image data or metadata".
Problems with various versions
DNG files with more than four channels? GroupBrowser
2007-04-15
Lawrence Taplin: "... the DNG SDK has kMaxSamplesPerPixel set at 4... Photoshop won't read other than the thumbnail".
Thomas Knoll: "... If there is a good reason, Adobe might lift the limit in its code"
Lawrence Taplin: I am working with several six-channel cameras designed for museum imaging ... It would be ideal to store the images in a publicly documented format that can be opened with standard software for colorimetric rendering".
Inconsistent DNG labeling: Metadata Browser Adobe Lightroom Forum
2007-04-22
Simon Iannelli: "DNGs I created with the multilanguage version of DNG converter are labeled as 'digital negative'. DNGs I created with the german version of Lighroom are labeled as 'digitales negativ' which is the german translation of 'digital negative'. Due to this the Metadata Browser has two categories for DNGs".
Unable to extract original raw file Adobe DNG Forum
2010-01-10
RicardoAT: "A year ago I started converter my ARW (sony raw) and after my NEF (nikon RAW) files to DNG with embedded raw, now I want to edit the dng in DxO pro 5 and it dosen´t read it, I try to get theoriginal raw and I only recovered as 60 percent of the original raws, thishappened to me in several batches of pictures and in different mac computers, I update the DNG converter 5.4 and is the same."
Problems with non-Adobe products
This section is not intended to be exhaustive. With over 200 non-Adobe products handling DNG files, inevitably there are bugs in some DNG-handling software! These are therefore selected because of their wider significance.
Apple Mac OS X v10.5.2,
Mac OS X Server v10.5.2

About Security Update 2008-002
(which cures this bug)

"A stack based buffer overflow exists in the handling of Adobe Digital Negative (DNG) image files. By enticing a user to open a maliciously crafted image file, an attacker may cause an unexpected application termination or arbitrary code execution. This update addresses the issue through improved validation of DNG image files".

What are some important considerations for workflows?

Keep in mind the importance of being resilient to any "single point of failure", such as loss of a disc or even loss of a whole computer.

Here are some (parts of) workflows (good and bad) for converting native raw files to DNGs, in order to illustrate some important issues. There are lots more workflows than these in use, and all photographers need to be comfortable with whatever they choose. For each workflow there are a number of steps, identified by the separate rows. These show copies, conversions, and deletions, of particular versions of the image files.

Step Capture Copy 1 Copy 2 Commentary
1 - A "single DNG" workflow
1 Native DNG  

This workflow probably appeals to people who enjoy parachuting without a reserve parachute! The native raw files are converted to DNG from the memory card, and then the memory cards are reformatted while there is just one copy of the DNGs.

It is vulnerable to disc failure, and there is no fall-back to the native raw file. But if the photographer didn't convert to DNG, a single copy of the native raws would still be vulnerable to disc failure!

2 (Delete) DNG  
2 - A "single DNG with embedded native raw" workflow
1 Native DNG(embed)  

Unlike "1", this has a fall-back to the (embedded) native raw file, but it is still vulnerable to disc failure. Although the DNG file may be a little smaller than the pair of copies in "3", it appears to have little merit.

Yet some people deduce that it is Adobe's recommended method, because Adobe have provided an option to embed the native raw file since the 2.4 DNG Converter.

2 (Delete) DNG(embed)  
3 - A combined "native raw" and "DNG" workflow
1 Native Native  

This is a popular workflow, offering considerable resilience. The native raw file is copied to a first disc as Copy 1. Copy 1 (not the original native raw file) is then converted to a second disc as Copy 2. Copy 2 is verified before the native raw file is deleted from the original memory. This latter verification verifies the integrity of Copy 1 by implication, because a verified Copy 2 is very unlikely to be possible from a bad Copy 1.
Work is done on Copy 2, and XMP metadata may be stored in it. Copy 1 remains available as a fall-back, although it may be migrated to optical media to release space on the first disc.
Kevin Connor (1 October 2004): " Nevertheless, we do recommend that customers using the Adobe DNG Converter archive two copies of their files: the original camera-specific raw file that contains all of the metadata and the DNG file that is more likely to be readable in years to come. When manufacturers support DNG natively, however, there will be no need to archive two copies".

2 Native Native DNG
3 (Delete) Native DNG
4 - A combined "DNG with embedded native raw" and "DNG" workflow
1 Native DNG(embed)  

This is a more sophisticated version of "3". It is for people who have more confidence in DNG, for example by assuming that it will be possible to extract the original raw file from the DNG file, and perhaps less confidence in their native raw format. It needs more storage space than "3".

Work is done on Copy 2, and XMP metadata may be stored in it. But also, XMP metadata may be stored in Copy 1, perhaps before it is converted to Copy 2, offering "asset management" advantages.

2 Native DNG(embed) DNG
3 (Delete) DNG(embed) DNG
5 - A "double DNG-only" workflow
1 Native DNG  

The native raw file is converted to a first disc as Copy 1. Copy 1 is then copied to a second disc as Copy 2. Copy 2 is verified before the native raw file is deleted from the original memory. This latter verification verifies the integrity of Copy 1 by implication, because a verified Copy 2 is very unlikely to be possible from a bad Copy 1. Work is done on Copy 2.

Compared with "3", this offers no fall-back to the native raw file. Therefore, it excludes the use of products that don't support DNG. And if the conversion to DNG loses some information, it is lost for ever once the native raw files are discarded. This should only be used by people who have sufficient knowledge to make an informed decision about the consequences.

2 Native DNG DNG
3 (Delete) DNG DNG

What are useful tests before adopting any upgrade to ACR or the DNG Converter?

Here are tests that can be used by anyone wondering whether to add DNG to their workflow, or before they adopt new versions of software, such as ACR or another raw converter or the DNG Converter.

You need to judge whether you are likely to use software that doesn't support those DNG files. If you sometimes want to do this, (and perhaps 58% of photographers sometimes choose to), you should keep your native raw files. But this need not stop you using a DNG-based workflow. Many photographers keep two copies of their image files in case of media failure, and many of those people keep their native raw files as one (archival) copy, and the DNG version as another (working) copy. Others use DNG with the native raw file embedded.

The most important question is: "does DNG make a practical difference to the image quality?", and these tests have pixel-perfection as their criterion. They apply whether or not you use ACR and/or Photoshop. Whenever Adobe release new versions of ACR and the DNG Converter I do these tests before replacing the previous versions for real work. If "Test 2" failed, I would not adopt the upgrade, and I would post the results to the Adobe DNG forum.

Test 1 - does the new version work as well as the previous version?

Sometimes an upgrade of the DNG Converter and/or the raw converter such as ACR may have minor bug clearances but no other significant advances. For any photographer who is already using DNG, and who wonders whether to take such an upgrade, here is a test that I use. Consider the case where the photographer has been using version 3.3, and wonders whether it is safe to upgrade to version 3.4. Use both of the following routes:

  • Native raw file > previous DNG Converter > DNG > previous raw converter > photo-editor
  • Native raw file > new DNG Converter > DNG > new raw converter > photo-editor

Then compare the results, for example as described below.

If they are identical, also use the full "Test 2". If they both work, the upgrade should not impact your photography. ("Test 1" typically showed differences when upgrading from 3.2 to 3.3, because ACR 3.3 changed the Bayer interpolation. It typically does not show a difference when upgrading from 3.3 to 3.4).

Test 2 - does the DNG route work properly?

Take some raw photographs. For each of them, use both of the following routes:

  • Native raw file > raw converter > photo-editor
  • Native raw file > DNG Converter > DNG > raw converter > photo-editor

Then compare the results, for example as described below. If they don't, post the results to the Adobe DNG forum.

How to compare two images for "pixel perfection"

You may be able to use the features of your photo-editor to check whether all of the pixels are identical in both cases. If you use Photoshop, a suitable check is:

  1. Have both versions of the photograph, (for example one that used DNG and one that didn't), open in Photoshop. Copy one of them into the other as a separate layer. (If you use "shift" while dragging, alignment will be accurate). Set the "Blending mode" of the upper layer to "Difference". (The resultant image will probably look black).
  2. Use the "Info" palette to see if at least one of the resultant pixels has the value 0 for all channels. (Obviously it is not possible to test all pixels this way!)
  3. Use Image > Trim, (with all the "Trim away" boxes ticked). This will try to trim the image according to the values of one of the corners. If it says "Could not trim because this would create an empty document" you know all the pixels have the same value, and you know this means 0 because "Info" has told you. So both versions are pixel-identical.

What are useful tests before reformatting memory cards after a shoot?

Before discarding your native raw files after converting them to DNG, or even simply backing-off your native raw files and making them less convenient to access, you need to know that you are unlikely to regret this! (My aim is to prevent any "single point of failure", such as loss of a disc or even loss of a whole computer, screwing me up. My biggest risk is probably loss or damage of the memory card before I transfer to my PC). These are not onerous checks. In part, they are a matter of doing things in the right order.

If you convert your native raw files to DNG before using them, verify that the DNG conversion has worked before making your native raw files less accessible or discarding them. Who knows what problems may arise in that path? Verify the output from that path, not the input. (Although note that if the DNG Converter encounters a file that has problems, it will throw an alert error in the Conversion Status dialog). For example, consider:

  • Native raw on memory card > native raw on safety disc > DNG Converter > DNG on working disc

If the output ("DNG on working disc") is OK, then both the input to the path (including the native raw files), and the path itself (including converting them to DNG), are almost certainly OK. Initial simple but useful checks are:

  • Ensure that final count of DNG files matches the original count of native raw files.
  • Visually check the sizes of the DNG files, for example in Windows Explorer, to ensure that they are "about" the right size. (For example, if your DNG files are normally about 5 megabytes, be cautious if any of them are less than 4 megabytes or greater than 6 megabytes).

Then, if you use Photoshop, look at the final folder(s) containing your new DNG files, using Browser (Photoshop CS), or Bridge (Photoshop CS2). These will read and examine those DNG files. If you see thumbnails, and especially if you can open those files in ACR, the conversion has been successful. You appear to have suitable DNG files converted from your native raw files, and any intermediate copies, for example for safety purposes, are also probably OK.


What do I (Barry Pearson) do?

I started to use DNG, still archiving the original raw files, (I use Pentax), on 10 October 2004. (Workflow 3). I started to convert directly from the memory card, and then discard the original raw files once I had two of each DNG file, on 20 June 2005. (Workflow 5). These have never caused me problems. I am not telling you to do this, and I will not be liable for any problems if you do so! You must make up your own mind.

Every day, I lose an infinite number of images. Perhaps I wasn't there, or didn't point my camera in the right direction, or didn't press the button at the right time. Even if I did all of those things, perhaps I didn't have the right camera, with the right lens, with the right settings. For an infinite number of images every day!

Anyone who is obsessed with every last bit of information for the infinitesimal fraction of available images they actually managed to capture has lost the plot! One of my biggest obstacles to delivering world-beating pictures is that I have diverted too many resources away from capturing new pictures towards playing safe with the ones I actually have captured. (That - and the fact that I am simply not a good enough photographer to deliver world-beating pictures!)