Benefits of DNG

Many of us who have been using DNG for years are amused (and puzzled) when we see people disputing the fact that there are benefits to DNG. I suspect that some of the critics have never even used DNG, or only used it for a brief period!

If we were a committee charged with deciding whether to develop a common raw format, we would need to exercise robust scepticism, and demand substantial evidence before committing the resources to develop the specification. But we are not such a committee, the resources have already been committed and spent by Adobe and others, and relevant practical evidence is widely available.

It is futile to debate whether any photographers can gain benefits from DNG when so many demonstrably do so. The debate needs to shift to: "what benefits are we talking about?"; "who can benefit?"; "is there an alternative?"; "what more needs to be done?"; "what evidence is needed to convince the sceptics?".


"your imaginary DNG benefits just show your sub-adequate skills in handling original files".
Julia Borg (2007-04-15)


Benefits for photographers

This section discusses the personal benefits that a photographer may get by using DNG. In other words, it attempts to answer the question "what's in it for me?" (A later section discusses the benefits of DNG for the industry as a whole, or for people, organisations, and companies other than the photographer).

Whether particular photographers can get any benefit from DNG depends on their workflow and the tools they use. (The situation gradually improves over time). Not everyone can get immediate benefit yet, or enough benefit to counter any perceived disadvantages. So any photographer who sees no current personal benefit in using DNG, and assumes therefore that there are no benefits to any other photographers, is wrong!


Tactical benefits

These are benefits that overcome some current technical issues, but are likely to become much less important or disappear entirely within years. They may currently be very important to individual photographers, but do not represent the future power of DNG.

Ability to use other software products

Some software products handle DNGs but not native raw files for certain cameras. One example is ACR 2.4 with Photoshop CS, which supports the same set of cameras as ACR 4.1 with Photoshop CS3, but most of them only via the DNG route. LightZone, Silkypix, GraphicConverter X/Classic, and IrfanView are other examples.

This is classified as "tactical" because individual advantages may disappear with time. (There is a corresponding "strategic" advantage, "Archiving", which will continue to have a long-term benefit). For example, a photographer who upgrades from Photoshop CS to Photoshop CS3 will no longer need this ability. This ability won't be needed by users of Sigma/Foveon and Silkypix if the latter ever supports X3Fs directly. However, the freezing of Pixmantec's RawShooter Premium, which wasn't such a product and so will never be able to support new cameras, reveals another benefit for users of such products if upgrades cease or are considered too expensive.

There is a niche use of DNG that may not have an alternative. Some cameras are not intended to have raw capability, but ingenious people have found how to extract raw image data from them! They then typically convert those raw files to DNG so that a range of products can process them. This won't benefit many people, but should be seen as a "proof of concept" of the power of DNG being a "self-contained" raw file format [raw2dng, smal2dng].

File size

Many photographers get significant benefit by exploiting DNG's compression, (JPEG lossless compression). Some people dismiss this, saying "storage is too cheap to worry about". But that doesn't allow for tactical problems such as people without DVD storage, or those who have limited high-speed connectivity for extra disc space. And while recent Canon and Nikon cameras tend to have well-compressed native raw files, several other manufacturers don't compress their native raw files at all.

Below are a few cases. All sizes are in megabytes. It is rare that either the native raw files or the DNGs for a camera have a consistent size, so these are simply examples. Where the native raw file is already DNG, it may still be smaller after re-conversion because of compression, but it will not be embedded by the "embed" option.

Sometimes the DNG file is larger than the original file. In the case of Sigma/Foveon X3F images, there is no direct FNG raw equivalent. In the case of Canon sRAW images, the SRAW image is in a hybrid format; it is not truly raw.

Camera Native raw file DNG (smallest) DNG (embedded)
Canon 10D CRW: 5.3 5.1 10.3
300D CRW: 6.0 4.2 10.0
350D CR2: 7.0 6.6 13.3
5D CR2: 13.7 11.3 24.8
Leica DMR back DNG: 19.3 14.3 None: (14.3)
D-Lux 2 RAW: 16.0 9.0 20.9
Nikon D1X NEF: 7.8 4.8 11.6
D2X NEF: 11.1 9.5 20.5
D70 NEF: 6.1 5.4 11.3
D200 NEF: 8.7 7.3 16.3
Pentax *istD PEF: 12.8 5.6 15.0
*istDL PEF: 9.9 5.3 13.2
Ricoh GR Digital DNG: 11.6 8.3 None: (8.3)
Sigma SD10 X3F: 5.2 Larger: 7.9 13.0
Sony F828 SRF: 16.5 9.0 25.5
R1 SR2: 20.0 11.9 28.3

"Recover edges"

There is a utility called "DNG Recover Edges" that can recover some pixels around the nominal sensor size for a camera. It operates on a DNG file, and changes the DefaultCropSize. This enables ACR, and any other raw converter that restricts itself to the DefaultCropSize, to see a larger image. (In future, if those raw converters allow those extra pixels to be used anyway, this utility will become redundant. In the meantime, it can often be useful).

DNG Profiles

Tools such as X-Rite's ColorChecker Passport and Adobe's DNG Profile Editor use DNG images in order to generate profiles for use in raw processors, especially (but not necessarily) Adobe's raw processors such as ACR and Lightroom. The resultant profiles can then also be used for processing the original raw images (as well as DNGs converted from those images) from the camera models concerned.

The reason for using DNG in these cases is that it avoids the need to update the tools as new cameras are launched. As long as someone, typically but not necessarily Adobe, can generate a DNG image from the camera model concerned, the tool can do its task of profiling the camera model, without needing to understand the raw file format for that camera model.


Strategic benefits

These are benefits that may be of immediate use to the photographer, although sometimes not. However, these benefits will accumulate over the years, and therefore are likely to appeal to photographers planning for the future. All of these exploit the fact that DNG files are openly-documented self-contained files, with a variety of metadata and other objects within them in addition to the raw image data.

Archiving

More about archiving on this site:

There is a need to be able to archive raw image files. This may be so that photographers and their descendents can retrieve images of personal importance in future, or so that images of wider historical importance can be retrieved. Archiving is not the same as backing-up, although there are some factors in common. A back-up is typically a reserve or substitute, often for relatively short-term resilience or performance reasons. After retrieval, it may be exploited in the same context as the original file, for example it may be accessed by the same software products. But an archive is intended to be exploited in a relatively unknowable context, for example where its origins may be in doubt, and by very different software products. (A back-up makes the image "now proof". An archive makes the image "future proof").

Some key requirements that DNG is intended to satisfy include:

"Future proof"?

  • Longevity and "critical mass". It obviously makes the archivists' task much easier to have relatively few formats to cater for, each able to cater for a large range of points of origin (such as cameras), over a long time. As noted elsewhere, DNG is designed to evolve and to cater for a vast range of camera and sensor characteristics and technologies.
  • An openly-documented self-contained format. Presumably the archivists' need for open documentation is obvious! "Self contained" means that the absolute minimum of additional information, other than the DNG file itself, is needed for the image to be retrieved. This especially means that separate knowledge of the characteristics of the camera that captured the image should not be needed, and certainly means that retrieval should not rely on specific software products from particular companies, Adobe or otherwise. DNG is intended to satisfy this, and each file contains the camera details - more than a colour profile. (Note that Adobe supply a DNG SDK comprising more than 100 C++ source files to optionally complement the documentation).
  • Comprehensive metadata (including preview). Ideally, any object of historical important should carry its own historical context with it. Many of the challenges faced by archeologists, historians, archivists, and librarians, arise from loss of the context for the object, or its separation from that context. See "Standard metadata" below. Obviously, it is important that the metadata itself is held in an openly-documented format, and this applies to XMP. (Note that Adobe supply an XMP SDK to optionally complement the documentation).

Standard metadata

A Metadata Manifesto

"A proposal by the Stock Artists Alliance for the adoption of guiding principles, standards and technology to promote image metadata use".

Dowload the PDF from:
A Metadata Manifesto

DNG can hold a considerable amount of metadata in XMP form. (XMP = eXtensible Metadata Platform). This metadata can be used by various products from Adobe and other companies for management purposes, and can also be copied into downstream Photoshop files, JPEGs, PDFs, and others. Examples of this metadata include:

  • "Rights management" metadata, such as IPTC's Creator, RightsUsageTerms, Instructions, CreatorContactInfo, and CopyrightNotice, and others such as WebStatement (website address).
  • "Asset management" metadata, such as keywords describing the photo-shoot, and describing the contents of individual photographs. This includes IPTC's Description, IntellectualGenre, Keywords, Location, Scene, and SubjectCode.
  • Editing and settings information of the raw converter, for example ACR can hold all such data as XMP within DNG files. (This is almost "tactical", because such information may become irrelevant with future software products. Only a few of these edits, such as "crop" and "align", are really software-independent).

Embedded previews

A DNG file can hold a number of JPEG previews. In DNGs created with ACR 3.4 or the 3.4 DNG Converter there can be up to 3 previews, with the following sizes: longest side = 256; longest side = 1024; and full image size. For option "Preview = None", the 1st is included. For option "Preview = Medium Size", the 1st and 2nd are included. For option "Preview = Full Size", all of them are included. Using the "Full Size" preview may make a compressed DNG file about 12% larger.

These JPEG previews are created by applying the editing and settings information of the raw converter and converting to sRGB, and so are equivalent to colour-managed JPEGs created as derivative files. Peter Krogh points out that large high-quality prints can be made from the full image size preview. These previews can be used by various viewers and asset management products, and in future will therefore help both with management and with reviewing settings. For many purposes, for example web galleries, quick prints, and email, extracting one of the previews will be as useful as (re)converting the raw image itself, and much faster.

(Specific details: "Full image size" uses DefaultScale but not BestQualityScale. "Crop" and "align" are taken into account. If the previews are created from ACR rather than the DNG Converter, the "Workflow options", including changing the resolution, are ignored. If the "full image size" preview is smaller than the "longest side = 1024" preview would be, the latter is omitted).


Benefits for the industry

This section discusses the benefits of DNG for the industry as a whole, or for people, organisations, and companies other than the photographer.


Better engineering

The downside of reverse engineering

Reverse engineering is the process of identifying the design of something by examining the thing itself, for example by analysing its structure or behaviour. In many case it is perfectly legal, and is often a necessary industrial task. Science itself is the process of reverse engineering the universe!

The "raw shooting" subset of the digital photography industry has exploited reverse engineering for several years. A camera manufacturer releases a new camera with raw capability, and probably supplies its own software to process those raw images files. Many other people and companies choose to compete with the camera manufacturers' software by supplying software that does a better job at processing those raw images, often but not always at a price. This whole industry is valuable for photographers. The problem is that the industry mostly continues to be based on reverse engineering rather than use of standard raw file formats! At first, before the value of 3rd party raw processing software was widely appreciated, this could be excused. It has now become a sign that the industry is still immature. It is shoddy engineering, wasting a lot of time and effort, leading to delays, errors, frustrations, and sometimes lower quality results!

This section identifies the damage that is being caused to the digital photography industry, and as a consequence to individual photographers. Too many people are complacent about the problems. Some even believe the spurious propositions that camera manufacturers' proprietary raw file formats support innovation, or that publishing raw file formats would give away commercial secrets. Those are the short-sighted tales of an industry in transition.

Some cautionary examples

This section only matters to people who believe that photographers should be able to use raw-handling software other than that supplied by their camera manufacturer. Those who believe it is OK to restrict photographers to the raw-handling software supplied by their camera manufacturers won't see what the problem is!

  1. (Ongoing) Delays and restrictions in supporting new cameras:
    When a new camera is launched, there is normally a delay until 3rd-party raw-handling software is upgraded with the details of the camera so that its raw files can be processed. Some such software may never support this particular camera if it isn't in the business interests of the software company concerned to do so. The delay occurs partly from the need to reverse engineer the raw file format in order to read it, but mainly from the need to reverse engineer some of the camera characteristics in order to process the data read from the file. (The full solution is to have an openly specified raw file format with camera details embedded. Partial solutions include documentation of camera manufacturers' raw file formats, and pre-disclosure of camera details for new models).
  2. (2005 & 2007) The dangers of a firmware download:
    A camera may output different raw files after a firmware change, and its files may then not be recognised by some raw-handling software. This illustrates that the code implemented as a result of the reverse engineering may be "fragile", and not take into account all possibilities. For the photographer concerned, this may make the camera virtually useless, although the camera manufacturer probably won't accept that. (In other cases, the firmware update may "only" disable one feature). This issue is related to "3" below.
  3. (2006) The dangers of an aberrant camera:
    A newly purchased camera may have an anomaly causing it to output raw files that are accepted by the camera manufacturer's software but not by some 3rd party products. For the photographer concerned, this may make the camera virtually useless, although the camera manufacturer probably won't accept that.
  4. (2006) Problems with tethered cameras:
    A camera may be supported via memory cards but not while tethered. Presumably the raw files have different formats depending on the method of ingesting them into the system. Sometimes it might be necessary to reverse engineer the raw files for a particular camera multiple times for different circumstances.
  5. (2006) Problems with raw files modified by the manufacturer's software:
    Compressing RAF files using Fujifilm HSV2, sometimes called HyperUtility, can make them unreadable to ACR. (And perhaps other raw converters). Thomas Knoll: "If you point to me some documentation on the compression format Fuji is using, I'll consider adding support. I'm not aware of any public documentation, and Fuji is not providing any private documentation (yes, Adobe asked)". This is related to "6" below.
  6. (2007) Windows Imaging Component codecs for Vista:
    Modifying NEFs using the 1st version of Nikon's WIC codec for Vista could prevent them being accessed by 3rd party raw converters. There may have been a similar problem using the default windows dowloader. "... file headers are modified in such a way that other editors ... are unable to open/recognise them". (An upgrade intended to fix this has been released). Modifying certain Canon raw files (those with extension "TIF" or "TIFF") using Microsoft software in Vista can result in complete loss of raw image data. As in "4": sometimes it might be necessary to reverse engineer the raw files for a particular camera multiple times for different circumstances.
  7. (2007) Canon WIC codecs for Vista:
    Apparently, although Canon have provided codecs that support CR2, these don't support CRW, which is the raw format for the D30, D60, 10D and 300D.
  8. Unsubstantiated rumours:
    Did Epson "silently" publish a firmware upgrade which changed their ERF RAW format, so that Adobe ACR could suddenly not open the files anymore?
  9. (2007) Canon 40D firmware incompatibility:
    The Canon 40D firmware v1.0.4 made the raw files incompatible with some raw converters such as ACR, Lightroom, and Silkypix, and with DNG Converter. (Firmware v1.0.3 and earlier versions were OK. Canon say they will fix the problem in later versions of the firmware). This is similar to "2" above.

These are not the sort of problems that only existed at an early stage of raw processing but no longer occur. Some of these are recent, and demonstrate that reverse engineering can still result in "fragile" code that fails with what the camera manufacturer may think of as minor changes to the raw file format. "1" is ongoing, and will continue until camera manufacturers switch to a common raw format such as DNG.

Flaws in the tales of the industry

Nikon's statement:

Nikon published an unconvincing justification for their "preservation" of NEF:
Nikon Advisory - April 22, 2005

Here are two assertions that some have attributed to camera manufacturers. These tales wouldn't matter if photographers were sufficiently sceptical. These tales don't stand up to scrutiny! But some photographers are influenced by the policies of their camera manufacturers. So it is worth pointing out that what is best for the camera manufacturers may not be best for their users!

  • "A common raw format would inhibit innovation":
    For some niche, innovative, products, the only way they will be supported by mainstream raw processors, whether from Adobe or otherwise, is to use a common raw format that doesn't require those raw processors to know about the product concerned. A common raw format doesn't automatically result in a "one size fits all" format of raw image data. DNG has features, disliked by purists but realistically necessary, to allow some "secret sauce" to be supplied by the camera manufacturer. It isn't necessary for high quality raw processing, but enables manufacturers to be innovative with features such as "picture styles" and "curves".
  • "Publishing our raw file format would reveal our commercial secrets":
    Within weeks of the release of a new camera with raw-capability, the essential details of its raw file format needed for high quality raw processing are known to many companies across the world! Then they get built into upgrades to a hundred or more raw processing products. The details necessary for high quality raw processing do not remain a secret for long! What benefit do the camera manufacturers get from trying in vain to keep that information secret? Some of the information in the raw files from those new cameras often does remain secret - typically (part of) the Exif Makernote. It would be much more sensible for the camera manufacturer to decide what it was determined to keep secret, for example its "secret sauce", and what it is willing to be widely known, and act accordingly.

Right from the start, DNG was designed to cater for these concerns of the camera manufacturers. That is why some camera manufacturers have begun to use DNG. Those other camera manufacturers' tales are false, and increasingly photographers and others are realising this. Change is uncomfortable, and a camera manufacturer's first project that uses DNG will be more expensive because of the "learning curve" involved. Other reasons for not using DNG are typically excuses.

Being informed

The primary purpose of these pages is to supply information to enable photographers, users of photographs, independent software developers, camera manufacturers, and librarians and other archivists, to make decisions that suit their purpose. The lesson is "be as well-informed as possible, subject everything you see to scrutiny, expect to see evidence for any claims, and accept that you are responsible for your own fate".

  • Be wary of the manufacturers' messages:
    This is self-explanatory. Subject them to scrutiny.
  • Identify what is good for photographers, not for companies:
    Some photographers appear unable to identify what they themselves would prefer in the absence of policies from their camera manufacturer. They appear to have an attitude what is best for our camera manufacturer is best for us, or a reluctance even to think about what they themselves would like. (This is especially true in Nikon forums. It isn't true in Pentax and Leica forums). The response should be obvious.

Support for niche variety and innovations

DNG is the raw file format of choice for niche and some minority camera makers.

If DNG didn't exist they would either have to use another camera maker's proprietary raw file frormat, assuming there is a suitable one, with all the dangers described above; or invent their iown, with the massive cost involved and the unlikelihood that there will be much software that can support it.

Instead, they simply use DNG, and lots of software products can support their cameras without even needing to know they exist. Obviously they are support by all relevant Adobe products, but lots more besides; the rest are typically not important enough to pose a handicap.


Reduced effort and reduced delays

This follows from much of the above. The total development cost and development time in the industry can be reduced by the existence of a common raw image format.

Just counting the number of camera makers that use DNG, and the number of camera models that write DNG, suggests that the proliferation of undocumented raw image formats has been reduced by perhaps 10 major types (10 file extensions) and perhaps 40 or so variations (different models within a type).


References