Commentary on "RAW format, the captive photo"

(Juergen Specht was asked to check this page for errors. His reply didn't identify any errors).


These interviews are at www.ladinamo.org (direct | via Internet Archive):

Javier García Diz: " The challenges of digital revolution in photography are driving camera and software manufacturers to make decisions that don't always agree with users interests. Maybe the most critical case is the one around the problems with the RAW image file format, whose lack of standarization could make impossible in time the use of millions of digital pictures. Ladinamo has talked with two of the people who have more to say about all this, Dave Coffin, software engineer from the States, creator of the ultra widely used dcraw program, and Juergen Specht, German photographer living in Tokyo and leader of the OpenRAW movement".

This page is a response to some parts of the article. The format is to use 3-column tables, with some of the questions and answers in the first two columns, and a response to those in the right column. Except where there are other specific points to address, the responses are concerned with "what do we do to achieve digital image preservation?"

Interview with Dave Coffin

Dave Coffin is a skilled and generous software engineer, justly acclaimed worldwide for the massive and ongoing contribution his dcraw has made to the raw shooting industry, based on his reverse engineering and software development skills. But can he make the transition needed for the maturity of the raw shooting industry, from use of reverse engineering to use of a common raw format?

Question from Ladinamo Dave Coffin's answer Commentary on that answer
What can the photographers of the world do in order to put pressure on manufacturers to build their RAW's open and documented? Instead of barking at the moon, they should work with me and Phil Harvey (the author of ExifTool) to reverse-engineer whatever they want to know about their raw photos.

At this stage in the lifecycle of the industry, reverse engineering is shoddy engineering. It is time to move on.
If this attitude had prevailed long ago, we would still be reverse engineering scanned file formats, and wouldn't have TIFF.

What is the reason for the manufacturers to be so closed with its raw specifications, structure and documentation? As I said above, you don't get rich by sharing all your trade secrets with your competitors. It is a myth that they would be sharing all their trade secrets. Within weeks of the release of a new camera with raw-capability, the essential details of its raw file format needed for high quality raw processing are known to many companies across the world!
Is the Adobe DNG format the pragmatic solution for raw problems or maybe the same old song? Neither -- it's another raw format that Photoshop and dcraw can read.

And how does that answer the question? It is a raw format that about 150 products from about 140 companies can read and/or write!

It's good for small camera makers who don't want to design and support their own raw formats. It is also good for camera manufacturers like Pentax who have already designed their own raw formats (such as PEF) and who now want to use what Pentax describe as a "General-purpose, publicly available RAW file format".
And it provides good color matrices for many cameras, something dcraw didn't have before.

It also provides lots of other camera details, enabling products to avoid having to build in those details.
Dave Coffin (1) (and here): "Not only is Adobe DNG now supported, the entire codepath has been redesigned for it. Adobe's XYZ->CAM matrices allow color science to replace black magic, whether decoding DNG or the original raw files".
Dave Coffin (2): "Adobe Digital Negative (DNG) is a great format -- I totally redesigned dcraw for maximum DNG compatibility".
Dave Coffin (3): "Adobe DNG is great for small camera makers who don't want the hassle of creating their own raw format and writing software to support it. And it's great for dcraw, because now I can do real color management with Adobe's matrices instead of my crude color manipulations prior to dcraw v7.00".

Interview with Juergen Specht

Juergen Specht was the main creator of OpenRAW. OpenRAW has blind spots about the causes and solutions of problems with camera manufacturers' raw files formats. His statements contain logical fallacies that help spread "fear, uncertainty, and doubt" about DNG.

Question from Ladinamo Juergen Specht's answer Commentary on that answer
Which is the main target of the OpenRAW movement? ... in some years it will be much much more problematic, because without a RAW documentation, programmers have little motivation to reverse engineer the RAW files from old, obsolete or unpopular cameras. There is no evidence that raw file documentation motivates programmers to support the raw files from old, obsolete or unpopular cameras! On the contrary, even supporters of OpenRAW choose which raw file formats to support based on the business case.
How do you think the OpenRAW.org site could help in this matter? We also reached the attention of camera makers who thought its ok to abandon support of older cameras and older RAW formats in their newer software and added this support later back, however sofar no camera maker uses a truly open documented RAW format.

There is little evidence that OpenRAW has reached the attention of camera makers. Sony made a noncommittal reply to the results of a survey, but that appears to be it!

But now some camera manufacturers do use DNG!

What is the reason for the manufacturers to be so closed with its raw specifications, structure and documentation? We can only speculate at this stage, but there might be a couple of reasons for their secrecy:
1) To create a vendor lock-in, so that you are required to buy the camera makers software for the camera makers hardware, adding more revenue from software.

Nikon and a few others sell their software.
But most camera manufacturers don't sell their software. Manufacturers know they can't require photographers to use their software.

2) Because if they are the only ones who see the specifications, they can change them whenever they want or invent something new. Camera manufacturers tend to evolve their formats fairly slowly. Open documentation wouldn't change that.
3) To conceal the extensive post processing (especially in terms of bad pixel count of their sensors) they do before they actually save the raw data as a RAW file. Specifying the raw file format wouldn't reveal the processing performed before they save the raw data as a raw file.
Is the Adobe DNG format the pragmatic solution or maybe the same old song? I wish it would be a solution, but Adobe decided for marketing reasons that they allow a backdoor to camera makers to store information in so called private tags, which remain undocumented. So even if some cameras can save DNG formats natively, some of the information a camera decides to conceal can be saved inside the DNG format and it becomes another undocumented RAW format after all.

Wrong! The openly documented information is sufficient to render high quality images. (Many products that support DNG use only the open parts of DNG files).
If a camera manufacturer stores private data, the DNG file simply becomes "a file with lots of documented data capable of rendering a high quality image, plus other stuff which may or may not be useful".

Plus Adobe stopped documenting their PDF and PSD format after it reached a certain market share ...

Adobe is systematically helping PDF become ISO standards.

DNG has similar status to TIFF - an openly specified file format owned by Adobe. Who worries about what Adobe may do to TIFF?
... so they can at any time release a DNG V2.0 format and decide not to document it.

No they can't! If it isn't documented, it isn't a DNG specification! From the license: ""DNG Specification" means any version of the Adobe Digital Negative (DNG) Specification made publicly available by Adobe".

No, DNG is unfortunately not a solution. That is simply a "blocking" statement to prevent examination of the ways that DNG is part of a solution.
What can the photographers of the world do in order to put pressure on manufacturers to build their RAW's open and documented?

Photographers should go ahead and tell camera makers at every occasion that they want that their RAW formats should be documented, because this would create a win-win-win situation for

* photographers - they can be certain that their RAW images survive into the future

Their raw image data will survive into the future, of course. What matters is whether they can conveniently be processed in future. There is no evidence that raw format documentation results in support of those raw files in future software.

* software developers - they can spend their time writing better innovative software than spending it reverse engineering new RAW files while being unsure if reverse engineering is lawful or not The problem is "proliferation of undocumented raw file formats". Lots of documented raw file formats is also a problem for software developers!
* camera makers - open documentation of RAW files would create trust among photographers and allow new imaging technology and usages created, camera makers would not even think of, thus increasing their market share. Use of a common raw format such as DNG is far more effective for this purpose!