Within weeks of camera launch, all the major raw converter suppliers have reverse-engineered the raw files and camera details enough to be able to perform high quality raw conversions. All essential aspects of the specification are known, and sometimes even published by 3rd parties.
Question: who do the camera manufacturers think they are hiding information from?
Answer: from 3rd party software companies whose mission is, in effect, to make the images from the camera look as good as possible!
How stupid can camera manufacturers be?
- How does DNG cater for innovation?
- Examples of DNG parameters for various cameras
- What choices do camera manufacturers have?
- Appendices
How does DNG cater for innovation?
Despite myths to the contrary, DNG does support innovation, and it accepts significant variations between camera models. It doesn't impose a "one size fits all" policy. (DNG is compatible with Adobe's ACR, which probably supports a greater variety of sensor configurations than any other mainstream raw converter).
DNG enables novel cameras to interwork with mainstream raw-handling software that might not see sufficient return on investment to support proprietary raw files from those cameras. Those cameras, or software supplied with them, can output DNG files, either the default form or Linear DNG.
DNG enables novel software products to process raw image data from a large set of cameras without the need to reverse-engineer their proprietary raw file formats. They can support DNG as a main file format, and a suitable DNG Converter, typically but not necessarily provided by Adobe, can be invoked, perhaps automatically, to create DNG files from raw files that the product can't read.
Background to DNG
DNG is based upon TIFF 6.0 and ISO standard TIFF/EP. TIFF/EP is also based upon upon TIFF 6.0. Some parameters identified here originated with DNG, for example "AntiAliasStrength". Other parameters originated with TIFF/EP, for example "CFAPattern". Yet more parameters, and the overall file structure, with its Image File Header and Image File Directories, are defined in TIFF 6.0 itself.
Terminology: This page talks about "DNG parameters", because the bit-level structure is not important here. Formally, these are TIFF fields, aka IFD entries, their names such as "AntiAliasStrength" or "CFAPattern" are TIFF tags, and the body of the table below illustrates TIFF values.
Evolution and "future proofing"
- DNG is designed to evolve. It has a version scheme built into it that allows the DNG specification, DNG writers, and DNG readers, to evolve at their own paces. (The specification is already on its 2nd version).
- The DNG parameters (tags) that define the camera details, describing colour and other characteristics of the camera, about 20 to 30 of them, will slowly be added to at a rate of perhaps one per year or two. Thomas Knoll once said that the last time DNG specification would have had to be revised as a result of new technologies was with the Fuji SR sensor. Each addition will open up lots of possibilities, for example the 2 or 3 parameters that cater for the Fuji SR sensors allow for a range of offset sensors, not just the Fuji version. Most new cameras are (at most) "state of the art", not "new art", and they fit within the existing parameters. (People might be surprised to learn how slow the rate of technology change actually is! The industry can only create and absorb new technologies at a low rate).
- DNG provides parameters (tags), (DNGPrivateData and MakernoteSafety), that allow manufacturers to "add value" to the raw image data and camera characteristics. This is in fact where the most rapid innovation exists. (For example, Nikon Curves and Canon Picture Styles are held in Makernotes in the NEF or CR2, and copied across to DNGPrivateData). Perhaps there will eventually be some degree of standardisation even here, but it is low priority compared with getting the raw image data and camera characteristics included in a standard format. And if/when there is standardisation, it will probably be easy to convert existing files to the new version.
- DNG can include XMP metadata within the file. X = "eXtensible". IPTC have published the XMP parameters that correspond to the old IIM parameters, including the Dublin Core. Adobe have made a start on the XMP equivalent of key Exif parameters, although I don't think this has yet been agreed with JEITA.
How DNG describes a camera's characteristics
DNG does a lot more than wrap raw image data in a common raw file format. It includes the camera details, describing colour and other characteristics of the camera, plus extra metadata, to enable unusual combinations of camera characteristics to be described.
Camera details comprise many parameters designed to record the technology differences between different cameras and their sensors. The parameters illustrated in the table below below are some of the parameters of DNG. They are not added at a high rate. Nearly all of them have existed since DNG was launched in September 2004. These are not separate parameters for different cameras. They are common parameters with different values for different cameras, so only brief extracts, from vastly more data, are shown here.
There are perhaps 20 or more such parameters. They can each have a number of individual values, and they can be combined in vast numbers of ways. Few cameras have the same values - there are just a few exceptions, such as the Nikon D70 and D70s, where the DNG parameters appear to be about the same.
The table below shows some extracts of parameters from DNG files for various cameras, showing that DNG's parameters can cater for some very different sensors and in-camera processing. (These extracts were made using the DNG SDK). There are also several parameters that are not illustrated below because of lack of space. DNG obviously also caters for such things as variations in numbers of pixels (horizontal & vertical), different bit-depths, different default crop sizes and origins, etc.
Examples of DNG parameters for various cameras
Linear DNG was not used for these examples. It hides some, but not all, of this variety, by demosaicing the image, turning it into a rectangular multi-colour image array, albeit less "cooked" than TIFF or similar formats.
| DNG parameter (tag) | ||||
|---|---|---|---|---|
| Support for less common sensor & CFA configurations: | ||||
| Nikon D1X (unsquare pixels) |
Sony F828 (4-colour CFA) |
Fuji SR2 Pro (staggered/offset array) |
Fuji SR3 Pro (staggered/offset array) |
|
CFARepeatPatternDim |
Rows = 2, Cols = 2 | Rows = 2, Cols = 2 | Rows = 2, Cols = 4 | Rows = 2, Cols = 4 |
CFAPattern |
Blue Green Green Red |
Red Cyan Green Blue |
Red Green Blue Green Blue Green Red Green |
Red Green Blue Green Blue Green Red Green |
CFAPlaneColor |
Red Green Blue | Red Green Blue Cyan | Red Green Blue | Red Green Blue |
CFALayout |
Rectangular (or square) layout | Rectangular (or square) layout | Staggered layout A: even columns are offset down by 1/2 row | Staggered layout A: even columns are offset down by 1/2 row |
SamplesPerPixel |
1 | 1 | 1 | 2 |
DefaultScale |
H = 0.7500 V = 1.5000 | H = 1.0000 V = 1.0000 | H = 0.7105 V = 1.4211 | H = 0.7149 V = 1.4298 |
BestQualityScale |
1.3333 | 1.0000 | 1.4074 | 1.3988 |
| Support for different strengths of anti-alias filter: | ||||
| Leica M8 & DMR back, Kodak DCS Pro SLR/c | Nikon D70, D70s, D50 | Virtually all other digital cameras, from most manufacturers | ||
AntiAliasStrength |
0.00 | 0.25 | 1.00 | |
| Support for different colour and tone mapping: | ||||
| Ricoh GR Digital | Nikon D200 | Canon 300D | Leaf Aptus 22 | |
ColorMatrix1 |
0.7724 -0.1822 0.0027 -0.5179 1.1992 0.3639 -0.1083 0.1894 0.7509 |
0.9483 -0.3451 -0.0142 -0.7374 1.4683 0.2943 -0.0723 0.0859 0.8369 |
0.8985 -0.3002 -0.0447 -0.5604 1.2978 0.2940 -0.1424 0.2167 0.9412 |
0.8035 0.1309 -0.1134 -0.6259 1.3352 0.3252 -0.1977 0.4003 0.6417 |
BlackLevel |
255.00 | 0.00 | 792.93 786.59 793.02 786.22 |
0.00 |
WhiteLevel |
4095 | 3880 | 32768 | 16191 |
BayerGreenSplit |
750 | 0 | 0 | 500 |
| And here are just a few examples of the other parameters (tags): | ||||
| Sony R1 | Nikon D2H | Sony F828 | Kodak DCS Pro SLR/c | |
BaselineExposure |
+0.00 | -0.75 | -0.25 | +0.25 |
BaselineNoise |
1.50 | 1.00 | 2.00 | 1.50 |
BaselineSharpness |
1.50 | 1.00 | 2.00 | 1.00 |
LinearResponseLimit |
1.00 | 1.00 | 0.80 | 1.00 |
| DNG's support for "within-model" variations: | ||||
| DNG demonstrably caters for variations between camera models, as shown on this page. But it goes further - it can cater for variations between different cameras of the same model. This enables a camera manufacturer, or the developer of a DNG Converter, to provide "fine-tuning" to suit an individual camera. Adobe's DNG Converter attempts to provide this information, where it can deduce it from information within the native raw file. Ideally, cameras themselves would supply this information. | ||||
| Sony R1 - camera X | Sony R1 - camera Y | OLYMPUS E-500 - camera X | OLYMPUS E-500 - camera Y | |
CameraCalibration1 |
0.9879 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 1.0191 |
1.0146 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.9689 |
0.8740 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.9717 |
0.9823 0.0000 0.0000 0.0000 1.00000.0000 0.0000 0.0000 0.9885 |
What choices do camera manufacturers have?
Camera manufacturers can use DNG in various ways to the benefit of users. Some ways affect the camera, some don't. All of these are freely allowed by Adobe's global royalty-free licence to use DNG. Camera manufacturers may provide combinations of one or more of these:
Usage of raw converters
- OpenRAW: "RAW Image File Conversion and Editing Software"
- OpenRAW: "Percentage of Respondents that Use Listed RAW Conversion/Editing Software Packages"
Pre-disclosure to Adobe
The Luminous Landscape: "... the Leica Digilux 2 is supported even though it wasn't shipping at the time that CR 2.1 became available. This goes to show you the benefits of manufacturers working closely with Adobe ... "
Thomas Knoll: "... Sony and Adobe worked together to get preliminary support for the A100 into Camera Raw 3.4 and the DNG Converter 3.4".
Thomas Knoll: "... there are other cases of camera makers working with Adobe".
- Use DNG as the camera's native raw format.
A few cameras and digital backs do this. DNG isn't (yet) be a suitable raw format for Sigma/Foveon cameras, but it can be used for other cameras, as illustrated on this page.
(Many "innovations" provided by cameras, such as Nikon's "Curves", and Canon's "Picture styles", are currently supported by Exif Makernote, and these can equally well be held inDNGPrivateData, or remain in Exif Makernote, qualified byMakerNoteSafety). - Use DNG as a camera's optional-extra raw format.
This sidesteps all doubts about whether DNG could support everything that the camera has to offer. (The Pentax K10D ws the first camera to do this). Everyone knows that JPEG can't do that either, but many people still choose to use JPEG in-camera, instead of raw. Why not let users decide for themselves?
(Remember - more photographers use Adobe's ACR than any other raw converter, and ACR uses exactly the same information that is held in a DNG file!) - Supply a DNG converter from the camera's non-DNG raw format.
(At the time of writing, I know of 10 non-Adobe DNG Converters, 4 of them provided by camera/back manufacturers so that their products will be supported by key mainstream raw converters).
Camera manufacturers know that if they don't do this, Adobe is likely to reverse-engineer their raw file formats and handle them anyway. So why not take control back from Adobe, and do it themselves? What can they lose?
(Companies with innovative sensor configurations could also supply a conversion to "Linear DNG" format, which would enable products that don't support that sensor configuration to handle the image. This would be the fastest way for any innovative sensor configuration to become established as a viable technology). - Ensure that all their software accepts DNGs for that company's cameras.
Camera manufacturers know that it will rapidly become possible to convert their proprietary raw files to DNG, and many of their users will do so. Why then exclude those users from their software?
(Hasselblad-Imacon accept DNG in cases where they don't use DNG as their native raw format). - Supply a high-quality converter from DNG back to the camera's own raw format.
This may sound bizarre - how does this help that camera manufacturer's users? It would enable users to have a DNG-based workflow without concern! A photographer could convert to DNG directly from the memory card, and discard the camera's original raw files, secure in the knowledge that the latter could be "reconstituted" if there was ever a need for the original raw file. It would remove one of the last inhibitions of some photographers to fully-adopting DNG. - Provide sufficient information to software companies to develop high-quality DNG converters.
Some camera manufacturers already do this, giving Adobe sufficient information under an NDA, (non-disclosure agreement), to have ACR and DNG support before the camera is released.
(In fact, they could conform to OpenRAW requirements for open documentation, and that itself would assist the development of better DNG converters).
Appendices
References
TIFF 6.0: Originally "Tag [or Tagged] Image File Format", revision 6.0. (Commonly known as TIFF or TIF). Owned by Adobe.
TIFF/EP: ISO standard 12234-2, "Photography - Electronic still picture imaging - Removable memory - Part 2: Image data format - TIFF/EP".
- Adobe source for the DNG specification
- Adobe - DNG SDK (Software Development Kit)
- Ross Finlayson, TIFF Information
- How often will DNG have to change for new cameras?
- ExifTool tag name documentation
IPTC - equivalent XMP parameters
Exif - equivalent XMP parameters
These were seen in extracts from DNG. (More are listed at ExifTool tag name documentation).
| Exif namespace | Exif AUX namespace |
|---|---|
| (http://ns.adobe.com/exif/1.0/) | (http://ns.adobe.com/exif/1.0/aux/) |
ApertureValue |
Firmware |
Examining DNG files
A convenient way of examining DNG files, for example to see the sort of information in them, is to use a small component from the DNG SDK. Specifically, (at least in the Windows version), there are 2 files that are useful: dng_validate.exe, and its documentation dng_validate.pdf. (In fact, there are 2 versions of dng_validate.exe, and the smaller "Release" version is suitable for this purpose, and runs faster than the "Debug" version). This utility was used for all the extracts shown on this page.
It is a DOS-like command-line utility, and a suitable command-line is:
dng_validate -v a.dng > b.txt
where "a.dng" is the name of the DNG file being example, and "b.txt" is the name of a text file which the utility is to create and use for the report.