This page discusses some topics related to raw shooting and raw files. It is not intended to be a comprehensive definition or discussion of the whole subject.
It includes some discussion of DNG, for example from minute-100 onwards.
What is in a raw file? Including a DNG file?
(Normal DNG files are raw files. They contain raw image data).
A typical raw file is highly structured internally, often (not always) using a tagged structure based on TIFF. Obviously the largest part of a raw file is the raw image data. But most of the individual tags are metadata of various sorts. The camera manufacturer's Makernote (or equivalent) tends to be a bit secret, but most of the metadata is pretty boring!
A common question is "what data (if any) is lost when converting a camera manufacturer's raw file to a DNG file?" (Does the Adobe DNG Converter discard raw image data? Does it discard any of that metadata? Does simply transforming the file format damage the image?) A little-asked question is "what data (if any) is added when converting a camera manufacturer's raw file to a DNG file?"
In the hope that much of the "fear, uncertainty, doubt" about DNG will dissipate when more is known about the contents of raw files, (including DNG files), this section reveals some answers to the above questions. A particular Nikon D70 NEF was compared with a DNG created from that NEF. Multiple tools were used "just in case" any one of them gave misleading results.
(Disclaimer: among the raw file formats that are handled with most integrity by the Adobe DNG Converter are NEF, CR2, and PEF. Some metadata problems may arise when converting some other raw file formats, including RAF, CRW, ORF, MOS, KDC. X3F files are handled very differently).
|Nikon D70 NEF||DNG from that NEF|
The NEF was downloaded from RAWpository. The DNG was created from it by the 4.1 DNG Converter. The DNG is uncompressed in order to simplify the following text a little.
_DSC4798.NEF downloaded from RAWpository; see "Firmware 2.00".
(NEF then renamed "NikonD70.NEF")
Uncompressed DNG created from _DSC4798.NEF by the 4.1 DNG Converter.
(DNG then renamed "NikonD70.dng")
The "verbose" outputs from "dng_validate.exe" from the DNG SDK.
In spite of its name, it is actually quite a useful "TIFF/EP" interpreter, as well as a DNG validator.
|In the above output by dng_validate.exe, the Makernote analysis starts most of the way down at:
In the NEF, this is in the EXIF Makernote tag itself. In the DNG, this is in the DNGPrivateData tag, which has a full copy of the original Makernote block.
"Main", "EXIF", and "XMP" tag listings from ExifTool. This is useful for examining what tags exist.
In the above output by ExifTool, the Makernote analysis starts most of the way down at:
"Main" and "EXIF" tag listings from ShowTags. This is useful for examining what tags exist.
(I added some blank lines for readability, but there are no other changes).
|Probably the only useful Makernote analysis is the length. The Makernote in the NEF is 32314 bytes, while the DNGPrivateData (which has a 20 byte header) is 32334 bytes, showing that all of the Makernote is copied.|
|NEF not DNG||DNG not NEF|
|Metadata tags in the NEF but not the DNG. (This is about tags, not about information).||Metadata tags in the DNG but not the NEF. (This is about tags, not about information).|
|These show the differences between the above tag listings from "dng_validate.exe" from the DNG SDK.
My comments in these files begin "**BP:".
|These show the differences between the above tag listings from ExifTool.
My comments in these files begin "**BP:".
|These show the differences between the above tag listings from ShowTags. (I replaced "unknown tag" messages with the real tag names from the TIFF/EP, EXIF, and DNG specifications).
My comments in these files begin "**BP:".
As far as these analysis tools are concerned, the DNG file is a superset of this NEF it was created from! They say that data is not lost by conversion to DNG - data is added to the DNG . (Except possibly for JPEGs embedded in the NEF by the camera).
This should not be a surprise. Since ACR 3.x and DNG Converter 3.x the entire Makernote has been copied to the DNG file. Then DNG Converter adds extra metadata to the DNG file as described at: Camera details embedded in DNG.
Informal definition of "common raw format"
I developed this definition for use by OpenRAW, and posted it to their website (2005-09-24).
This is an informal, yet practical, definition of "common raw format". It simply expands on "common", "raw", and "format", avoiding arbitrary rules.
- It is a file format capable of holding what camera makers consider to be raw sensor data. It is not some alternative to a file format, such as an application programming interface, or a software or hardware product such as a software development kit.
- It is openly specified and freely licensed, so that it can be read and/or written by lots of software and hardware products of various kinds from various makers. File writers may at least include digital cameras of various types, file converters, and raw converters. File readers may at least include raw converters, image viewers, and asset management systems.
- A file reader that has implemented the full specification is capable of processing files from various file writers without having to be designed for those specific writers. A file writer does not have to be designed for specific readers. So the format is self-contained, at least in the sense of not relying on things that are not themselves common file formats.
- The specification of the file format can evolve to cater for new products that have characteristics not previous supported. But a common raw file does not cease to be "common" when it does not conform to the latest version of the format. So the format has a version control scheme so that file readers, file writers, and the specification itself, can evolve at their own rates.
- A raw file that contains data that is not openly specified may still be a common raw file if that data can safely be ignored. It is a common raw file if and only if the raw file would be a common raw file if that data were discarded, as long as the open specification is sufficient to enable that data to be ignored or discarded.
What is a "Digital Negative" (if anything)?
Even before Adobe chose the name "Digital Negative" (and DNG) for their file format, the term "digital negative" was in common use. The context was often "your raw files are equivalent to your negatives from a film camera, and so your raw files must be preserved". This is erroneous - they are not equivalent to negatives from a film camera, and they do not necessarily have to be preserved in their original form!
Why not to call it what it is - "digital latent image"?
- A raw file contains both "raw image data" and lots of other data, the latter often generically called "metadata". If anything resembles the negative from a film camera, it is the raw image data, not the whole raw file. But even that stretches the analogy a bit ....
- The raw image data needs to be processed in an un-prescribed way before it can be exploited as a digital image in typical applications. In that sense it is less processed than a developed negative from a film camera. It is more like the undeveloped negative, although with the quibble that it isn't negative!
- "Raw image data" itself is often erroneously stated to be the data that comes from the sensor of a digital camera. In fact there is significant processing needed to turn the output from a sensor into the digits that get stored in a file, and the amount of processing can vary from one camera to another. "Raw image data" means what the camera manufacturer chooses it to mean, typically to distinguish it from more processed ("cooked") forms of image data such as TIFF and JPEG.
Does this matter? In some circumstances, yes it matters!
- I believe it is patronising to tell people erroneous things in order to simplify the explanation. They may have difficulty revising their beliefs later.
- It can get in the way of some tasks that people may want to perform later. Conversion to DNG typically preserves the raw image data, and therefore can be useful as a way of preserving the digital latent image. A number of software products change the metadata held in raw files while preserving the raw image data. These actions conflict with any assumption that raw files must be preserved. Such products are increasing in number - management of metadata is increasingly important.
Why did Adobe choose the name "Digital Negative"? I don't know, but I assume it was thought to be a useful brand that would help DNG become established.