OpenRAW and DNG

(Juergen Specht was asked to check this page for errors. His reply didn't identify any errors).



What is OpenRAW?

OpenRAW is an organisation (with a web-site) that lobbies for camera manufacturers to publish their raw file formats. It was launched towards the end of April 2005 with the motto "Digital Image Preservation Through Open Documentation". During the first part of 2006 OpenRAW conducted a survey of the views of photographers towards the problems of the proliferation of unpublished raw formats. About 19,000 people responded, and OpenRAW intends to use the results in its campaign. OpenRAW is right to oppose the ongoing need for reverse engineering of raw file formats from camera manufacturers.

The period after the launch, and the period of the survey, were peaks of high visibility for OpenRAW. It could have exploited these to add weight to the sort of practical things that photographers urgently needed. Perhaps it could have been effective at publicising the need for a common raw format as part of the solution to the problems that OpenRAW articulated. Since DNG was about 7 months old when OpenRAW was launched, OpenRAW could have identified DNG as such a format, and built it into its strategy somehow. But initially it did its best to ignore DNG, then later disparaged it.


OpenRAW's blind spots

OpenRAW lacked important knowledge, and also ignored problems that were inconvenient to its narrow aims.

1 - lack of knowledge of the archiving discipline

DNG is the only raw file format specifically designed, inter alia, to be used as an archival format.

One of its metadata formats, XMP, is used in an archival standard, ISO 19005-1 ("PDF/A").

DNG is gradually gaining recognition for archival purposes.

Although OpenRAW has a motto containing "Digital Image Preservation", it appears never to have understood the discipline of digital resource preservation. Obviously there is no reason why OpenRAW should lobby for all of the features needed to achieve "Digital Image Preservation". But it should cooperate with a coherent "architecture" for the topic, so that its efforts will work with, rather than against, all the other things that have to be done.

The wider topic of digital resource preservation, and the sub-topic of digital image preservation, have a much longer history than OpenRAW or even DNG. There is an ISO standard for the architecture (called OAIS): ISO 14721:2003, "Open archival information system - Reference model". Many academics worldwide are researching the sub-sub-topic of archiving raw image data. Some of them are waiting to see if DNG becomes the standard format for archiving raw files. Archivists are necessarily conservative people (in all senses!)

The OpenRAW statement "Open documentation of all RAW file formats by manufacturers is the quickest and most satisfactory way for OpenRAW's goals to be reached" is naive and inadequate:

  • There is no evidence to support that statement, for example no evidence that their goals would be reached!
  • When planning for decades ahead, pursuing the "quickest" way is less important than pursuing the most robust and comprehensive way, after research to determine what that is.

Images are not preserved if they can't be found, or can't be identified when they are found. OpenRAW has never satisfactorily addressed the linking, and preferably embedding, of metadata for asset management, rights management, or rendering decisions. (This is covered by the more general term preservation metadata).

2 - what about camera details?

DNG holds camera details within the file itself.


From John Beardsworth:

Apparently Nikon and Canon will open up, include discontinued cameras, be joined by new DSLR makers like Sony and Samsung, and those manufacturers who've already withdrawn from the DSLR market. Really?

OpenRAW's emphasis has been on documentation of the raw file format. But that is insufficient by itself. Full documentation of the format is not enough for a raw converter to render a scene accurately.

A raw converter needs sufficient knowledge of how the camera generates raw image data from the image. For example, which colours and tones in the scene correspond to particular sensor values? Raw converter developers discover this by obtaining a camera and taking test shots, matching known targets with raw image data values. They then build that information into their product and apply it to raw image data from that model of camera.

In future, where will raw converter developers get that information for today's cameras? They may not have a camera of that model to test. What is needed is documentation of those camera details, in a form that can be used by future raw converters. There needs to be a defined set of information about camera models that is obtained when the camera is released. Where is the definition of what information will be needed?

Contrary to popular belief, a camera manufacturer's raw file format typically doesn't change much from one of its cameras to the next. For example, NEFs have similar formats across lots of camera models. (Top end cameras have GPS EXIF data. Over time, the Makernote gets more fields within it). But the format is not the main problem - information about camera details is a bigger problem. Ideally, those camera details would be held in the raw file itself.

3 - what about proliferation?

DNG is a common raw format. It will reduce the problem of proliferation.


From John Beardsworth:

And then what? Will the imaging program that you decide to use, routinely or simply one day as a promising trial, fully utilize your particular camera's raw format secrets? Really?

The problem identified by OpenRAW has two main components, of which OpenRAW is concentrating only on the second. It is caused by a proliferation of undocumented formats. "The Raw Flaw" was never just about undocumented formats. It was also about the proliferation of camera models, and the low probability that future raw processing software would bother to cater for all of them even if they were all documented.

The first article posted to the OpenRAW web site, by Mario Westphal, was at least as much about proliferation as about lack of documentation. In 10 years time, if we have documented raw formats, but far more of them, OpenRAW's goals won't have been reached. Proliferation is an unacceptable problem in its right, and turns what would be the inconvenience of just a few undocumented formats into a serious ongoing problem.

Obviously, this problem has to be tackled by reducing variety, for example by using a common raw format. Then future raw processing software can support that format and hence lots of camera models, including some for which there wouldn't otherwise be a business case. Dozens of new cameras with raw capability are launched per year. They don't all have different file formats, but they typically need different camera profile information in raw converters. New raw handling software products are steadily appearing, sometimes by adding raw support to an existing product, sometimes by introducing a new product.

When those software products support raw formats via new code they tend to support the most recent cameras first, and add other cameras later, if at all. Will a photographer's future tools of choice support today's cameras? It would be possible to add such support to those tools if they don't - but will that actually get done? And at what cost? OpenRAW never analysed this.


Early antipathy towards DNG

For about 4 months from its launch at the end of April 2005, OpenRAW had high visibility. It could have exploited this to publicise the need for a common raw format. This could have been as well as lobbying for open documentation of camera manufacturers' raw file formats.

Downplaying the need for a common raw format

Why didn't OpenRAW throw its weight behind the importance of establishing a common raw format, then compare DNG with the aims of such a format, so that the development of DNG could be judged?

The originators of OpenRAW were cool towards towards DNG during this period. It wasn't mentioned by name at the time, and it was dismissed as not satisfying OpenRAW's objectives. Cameras that use DNG as their native raw format (the first appearing in June 2005, about one and half months after the launch) have not been listed as supporters of OpenRAW's aims, even though DNG is openly documented.

There is no doubt that any use of a common raw format as defined here would satisfy OpenRAW. This definition was specifically designed to conform to OpenRAW's objectives, and posted to their website. That is not to say it should be the only approach - but why isn't it one approach adopted by OpenRAW?

Omissions from "The Raw Flaw"

OpenRAW once prominently displayed a document called "The RAW Flaw". The authors were Michael Reichmann (of Luminous Landscape) and Juergen Specht (of OpenRAW). The document was published on both sites with differences.

The Luminous Landscape version said:

"The Solution? There really is only one solution – the adoption by the camera industry of...
A. Public documentation of RAW formats; past, present and future
or, more likely...
B. Adoption of a universal RAW format"

The OpenRAW version omitted the "or, more likely...". This increases the emphasis on public documentation, and reduces the emphasis on a universal RAW format. Another omission from the OpenRAW version were the sentences:

"Finally, consider the problems of digital asset management and the cataloging of files. Many pros as well as amateurs have literally Gigabytes of RAW files. The various asset management programs can't hope to keep up with the ever increasing number of proprietary formats. And as time passes and these programs are enhanced, what are the chances that they will still be able to read your older RAW files?"

That paragraph made the point that it isn't sufficient to have public documentation, but it is also important to have a universal RAW format to reduce the proliferation. Presumably this was not a message that OpenRAW wanted to give.

Other parts of the page (on both sites) do mention the use of a a universal RAW format, and mention DNG by name. That page is no longer prominently displayed at OpenRAW. People who didn't know it was there would probably not find it. The concepts of "a universal RAW file format" and "Adobe has put forward the DNG format as an open standard" have been hidden by OpenRAW, presumably because they are inconvenient for OpenRAW's own narrow objectives.


Later hostility towards DNG

OpenRAW had another period of high visibility during the period of the survey and its publication, for about 4 months from end January 2006. Instead of supporting the only credible progamme that was delivering tangible benefits to photographers, its attitude towards DNG became more hostile, and this has spread "fear, uncertainty, and doubt" in some forums since then.

Airbrushing DNG away

Up to end March 2006, the OpenRAW web site said "Many have suggested (and Adobe has created) a common, open file format for RAW image files for all camera makers to use as a solution to the RAW problem". (Note that it didn't mention DNG by name!) From April 2006 it omitted "(and Adobe has created)". In effect, this retracted their 11-month position that DNG is indeed a common, open file format.

The document "The Raw Flaw", which already downplayed the need for a common raw format compared with the Luminous Landscape version, ceased to be as visible. This virtually removed the concept of a common raw format, and the name "DNG", from any visibility.

Screen-shot of part of the OpenRAW home page

Explicit anti-DNG views

Soon after that, OpenRAW prominently published a large article "DNG is not the answer" by Stuart Nixon. Juergen Specht had said "I will publish Stuart's mail regarding DNG etc. as article, he gave me the permission". This article was edited and renamed "Notes on the future of Open RAW formats, and a look at DNG" a few days later. The theme remained the same, including the words "DNG IS NOT THE ANSWER". The way Juergen Specht chose initially to display the article in that manner suggested hostility.

That article has been much-quoted in forums, with no attempt at a retraction by OpenRAW or Stuart Nixon. It has promoted "fear, uncertainty, and doubt" about DNG, using the "OpenRAW" brand, in spite of the fact that it doesn't stand up to scrutiny.

Recent ongoing hostility towards DNG

More recently, Juergen Specht has been quoted as responding to "Is the Adobe DNG format the pragmatic solution or maybe the same old song?":

"I wish it would be a solution, but Adobe decided for marketing reasons that they allow a backdoor to camera makers to store information in so called private tags, which remain undocumented. So even if some cameras can save DNG formats natively, some of the information a camera decides to conceal can be saved inside the DNG format and it becomes another undocumented RAW format after all. Plus Adobe stopped documenting their PDF and PSD format after it reached a certain market share, so they can at any time release a DNG V2.0 format and decide not to document it. No, DNG is unfortunately not a solution".

That is largely false! In fact:

  • The openly documented information is sufficient to render high quality images. (Many products that support DNG use only the open parts of DNG files). If a camera manufacturer stores private data, the DNG file simply becomes "a file with documented data capable of rendering a high quality image, plus other stuff which may or may not be useful".
  • Adobe is systematically helping PDF become ISO standards.
  • If it isn't documented, it isn't a DNG specification! From the license: ""DNG Specification" means any version of the Adobe Digital Negative (DNG) Specification made publicly available by Adobe".

Creating "fear, uncertainty, and doubt" using fallacies

A typical example is calling DNG "a lossy format" because some metadata for minority cameras is probably not (yet) copied across, even though it is copied across for Canon, Nikon, and Pentax, etc. Another is to falsely claim that because camera manufacturers don't document their Makernotes, this information can't be made available in the DNG file. Here are known fallacies used by OpenRAW:

Peter Krogh makes the point:

I actually think that the metadata people create once they start working with images will be of significantly more value that the relatively small amount of information hidden in the private maker notes. You will have a lot more vested in your ACR, (or Aperture, or C1 or Bibble, or Silkypix, RSP, Lightroom) settings than you will in knowing how the camera remapped dark pixels....

If this stuff [the small amount of metadata that is contained in Private Makernotes] was absolutely necessary for a great (or even good) conversion, then C1 Pro would not be able to beat manufacturer's software.

  • Failure to distinguish between a lossy format and lossy use of a lossless format:
    DNG is a file format specification. It is capable of lossless storage of raw image data and metadata. (Perhaps with the exception of Sigma/Foveon cameras). Some uses of it may be be lossy, but that doesn't make it a lossy format.
  • Failure to distinguish between some lossy uses and and majority lossless uses:
    Adobe claims, and OpenRAW doesn't refute, that DNG conversions of NEFs, CR2s, PEFs, and possibly some others, are lossless, both for image data and metadata. That probably covers the vast majority of raw shooters in the world, for whom DNG is lossless both as a format and in practice. Those photographers need to be reassured, not worried. And it is a proof of concept for other cameras.
  • Failure to distinguish between important data and unimportant data:
    This is illustrated by Peter Krogh's point. Even for camera models where the DNG Converter may not (currently) copy across all the metadata from the original raw file, it copies across the raw image data. (OpenRAW is concerned with "Digital Image Preservation", not "Metadata Preservation").
    Imagine the future scene: "yes, we have the picture, we know who took it and what it is about, and we can render it as the photographer chose - but it is useless, because we don't know what lens was used"!
  • Failure to distinguish between lost information and undocumented information:
    Some of the metadata in the camera manufacturers' raw file formats is undocumented. When it is copied across to DNG files, it may remain undocumented, (although not always). It hasn't become lost - in fact, it hasn't changed its status. It simply awaits explanation, whether it is in the original raw file, or in the DNG file. Obviously, DNG can't be criticised for that!
  • Failure to distinguish between "what is done now" and "what can be done in future":
    The DNG specification is evolving, and so are software products that exploit it. At the time of writing, (2007-02-09), the specification is at the 2nd version, with a 3rd version to come within months. The 2nd version can hold more data than the first. The 3.x versions of Adobe's products preserve more metadata than the 2.x versions. Who knows what can be achieved in future?
    If there were a better alternative, it may be sensible to say "why bother with DNG?" But there isn't another contender for an archival raw file format. (Professional archivists won't adopt NEF or CR2, etc). Why isn't OpenRAW helping to turn DNG into its ideal mechanism for "Digital Image Preservation Through Open Documentation"? (Or perhaps DNG is that already. OpenRAW hasn't proved otherwise).
  • Failure to judge alternatives by the same criteria:
    Alternatives to DNG are never ideal formats themselves. What they typically have in their favour is "inertia" - photographers may simply be using them by default, and never have subjected them to scrutiny.

Here is an article that appears to agree with these points:

In my view, the purpose of the two initiations have several common points. DNG has a versioning system, and maybe if members of openRAW would suggest alterations on DNG, then the next release of DNG (if Adobe would have implemented theese theoretical alterations) could be "the" openRAW. DNG is very well accepted, and together, Adobe and openRaw would be able to persuade more manufacturers to implement a common standard.

How does this affect DNG?

The OpenRAW group may consider that OpenRAW and DNG are competitors, or are incompatible, or are independent of one-another. In fact, OpenRAW is really a facilitator for DNG, whether the OpenRAW group want this or not! It would be helpful if the OpenRAW group realised that DNG addresses problems that OpenRAW needs solutions for, and that OpenRAW can't succeed without helping the take-up of DNG.

DNG doesn't need OpenRAW, although it helps. Despite denials, OpenRAW really needs a common raw format, for which DNG is the only contender. And perhaps 200 or more of OpenRAW's own individual supporters have expressed a desire for a common raw format, often mentioning DNG by name. Fortunately, if OpenRAW succeeds in getting the cameras' raw file formats published, this will inevitably accelerate the progress of DNG.

In summary:

  • The preservation problem that OpenRAW is rightly concerned about can be solved by a combination of all relevant software fully supporting DNG, and all raw files being fully convertible into DNG.
  • Openly-published raw file formats will help DNG evolve in a sound manner, enable better DNG converters to be developed if necessary, and give people more confidence that they can rely on DNG.
  • But there is no evidence that openly documenting raw file formats will increase the likelihood of their support by software products. There has been no rush by raw converter developers to support the cameras and digital backs that use openly-published DNG as their native raw format!

Mentions in DPReview forums

OpenRAW posts per day on DPReview since launch: fewer than 0.6.

DNG posts per day on DPReview since launch: more that 15.

Activity on the OpenRAW website itself at June 2007

Link to OpenRAW threads statistics

References to OpenRAW in blogs at 11 February 2007

Link to OpenRAW blog statistics

References to OpenRAW in blogs (9 in 180 days) at 14 April 2008

Can OpenRAW ever become effective?

It is likely that OpenRAW is now in terminal decay.

There is now little reference to OpenRAW in newsgroups, blogs, and forums. The OpenRAW website is mostly comatose. There is no evidence that OpenRAW ever caused a single camera manufacturer to document its raw file format. OpenRAW's success was to make many people aware of some major problems. Its failure was to identify an inadequate solution, and hence not to improve the situation.

While DNG can provide tangible immediate and future benefits to photographers, OpenRAW can only offer "best endeavours". It only achieves high visibility (important to a lobby organisation) when it has something novel to announce.


OpenRAW needs a new initiative, and perhaps exploiting the undoubted existence and progress of DNG would resuscitate it. (It would hit the headlines - what else will do that?) Suggested methods include all of:

  • Emphasise the need to reduce proliferation as well as achieving open documentation. Hence promote the requirement for a common raw format, and publish a practical definition. Restore the missing bits to "The Raw Flaw".
  • Analyse and evaluate DNG against the OpenRAW definition of a common raw format, identify the circumstances in which it conforms, and determine what needs to be done in other cases. Publish the results for debate.
  • Identify and publish where OpenRAW's activities fit within the global research and development in digital resource preservation, digital image preservation, and raw file preservation. Publish a corresponding action plan. (Start here).
  • Tell camera manufacturers that an acceptable alternative to publishing their native raw file formats is to provide software to convert them to an openly documented raw file format. For example, providing a DNG converter from their own formats to DNG, together with documentation of DNGPrivateData, satisfies the objectives of OpenRAW, because the end-result is a fully documented raw file. (This is an approach that has been used by Pentax, Hasselblad-Imacon, and Better Light, and so there is evidence that it can work).
  • Publish guidelines for photographers who shoot raw, to help them take into account the undoubted benefits of DNG, the issues of unreliable media, the current known problems with DNG in some circumstances, and possible future trends. This may be modelled upon: SaveMyMemories.org, or the Universal Photographic Digital Imaging Guidelines. (Or simply contribute material to those websites).

Conclusion

OpenRAW is a "single issue lobby group". Such groups suffer from the "law of unexpected consequences". They assume that if they cause a particular change, other things won't change to thwart their aims. OpenRAW appears to assume that if they cause raw file formats to be documented so that raw converters have better information to work on, the result will simply be that those raw converters will better support those formats. That isn't how things work in the real world! Human nature plays its part. Other players exploit the results.

What will happen with full documentation of raw file formats? One obvious probability is that Adobe and others will develop better DNG converters. What will happen when there are demonstrably better DNG converters? Obviously people will have more confidence in DNG, and more people will adopt DNG-based workflows. Will those future raw converters support several hundred cameras simply because their raw file formats are documented? Experience so far suggests not. But with more people adopting DNG-based workflows, those raw converters will have to support DNG.

OpenRAW were doing the right thing, except for their hostility towards DNG. And if they succeeded with their lobbying, the world of "raw shooting" would transform faster into the world of "DNG shooting". That would be good for the future health of top-end digital photography.