Photo Forum / Digital Photography / DSLR Cameras / May 2005
"Raw" file issues?
|
|
Thread rating:  |
RichA - 26 May 2005 17:31 GMT http://www.luminous-landscape.com/essays/raw-flaw.shtml
eawckyegcy@yahoo.com - 26 May 2005 19:07 GMT RichA trolls:
> http://www.luminous-landscape.com/essays/raw-flaw.shtml http://en.wikipedia.org/wiki/Internet_troll
Some day I will expand on this entry, by noting that "trolling" behaviours are hardly limited to the Internet -- computer networks did not magically allow people to suddenly make straw-man arguments. Consider the classic example of a politician -- our polite name for those who fabricate false arguments/statements as a professional activity -- or even the media itself who regularly troll their audience with prurient, irrelevant, or just plain false information.
Alan Browne - 26 May 2005 20:29 GMT > activity -- or even the media itself who regularly troll their audience > with prurient, irrelevant, or just plain false information. Don't be an a.s. Given the indications that companies like Nikon would like nothing better than to sell more s/w as aprt of their camera solution (other OEM's too), everyone has everything to gain by a standard for RAW images.
 Signature -- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm -- r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm -- [SI] gallery & rulz: http://www.pbase.com/shootin -- e-meil: Remove FreeLunch.
eawckyegcy@yahoo.com - 26 May 2005 21:10 GMT > > activity -- or even the media itself who regularly troll their audience > > with prurient, irrelevant, or just plain false information. [quoted text clipped - 3 lines] > solution (other OEM's too), everyone has everything to gain by a > standard for RAW images. It is not in Nikon's interest to tell everyone their innovations in (say) automatic white balance, or in Canon's interest to spill their beans re: (say) fancy device physics for optimal bias estimation in long exposure images.
Reichman, et al, say they don't want "trade secrets" revealed, but they are woefully clueless in that their demands for documentation of these files is exactly that.
If you don't like Nikon's or Canon's policies about any of this, you are free to purchase the products of other companies. Right? Or is someone forcing you to purchase Nikon's software?
Really, what exactly is the problem here? If anything, the very existance of Dave Coffin's "dcraw" makes the ranting Reichmann, et al, look very kookish, and rendering the entire "OpenRaw" issue moot: RAW file formats are as open as can be, _despite_ the best efforts of Nikon.
Alan Browne - 26 May 2005 21:26 GMT > It is not in Nikon's interest to tell everyone their innovations in > (say) automatic white balance, or in Canon's interest to spill their > beans re: (say) fancy device physics for optimal bias estimation in > long exposure images. No. They give away all the data to "qualified" software developers. They are not protecting anything to a degree where it would remain a secret for more that a week or so. That it can be cracked easilly by writers of sw such as dcraw are testimony to the fact that it is pointless.
> If you don't like Nikon's or Canon's policies about any of this, you > are free to purchase the products of other companies. Right? Or is > someone forcing you to purchase Nikon's software? Yes, Nikon. For a person with 000's of dollars in Nikon glass, being forced to purchase more beyond a camera is simple wallet gouging.
> Really, what exactly is the problem here? If anything, the very > existance of Dave Coffin's "dcraw" makes the ranting Reichmann, et al, > look very kookish, and rendering the entire "OpenRaw" issue moot: RAW > file formats are as open as can be, _despite_ the best efforts of > Nikon. The point is to stop these idiotic and pointless 'races' in the future. When you buy a film camera, it works with the specified film. No special to Nikon processes are required. Nikon are now requiring special to type processes for their digital cameras.
 Signature -- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm -- r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm -- [SI] gallery & rulz: http://www.pbase.com/shootin -- e-meil: Remove FreeLunch.
DoN. Nichols - 27 May 2005 03:20 GMT [ ... ]
>> Really, what exactly is the problem here? If anything, the very >> existance of Dave Coffin's "dcraw" makes the ranting Reichmann, et al, [quoted text clipped - 6 lines] >special to Nikon processes are required. Nikon are now requiring >special to type processes for their digital cameras. When you buy a film camera, you are locked into the physical format of the film. If you want to shoot on some Royal-X Pan (very high speed B&W film -- probably now long dead), it was available in 120 and 620 formats, but *not* in 35mm. (For good reason, as the grain would have been excessive in 35mm.)
Using 120 roll film or 4x5 sheet film in a 35mm camera is not practical (though there were some strange adaptors in existence, including one to take a 35mm image from the back of a Nikon F and expand it to expose a 4x5 sheet film, or a 4x5 Polaroid (usually 3000 ASA).
And to use other films, such as Infrared Ektachrome in your 35mm SLR, you have to put some rather extreme filtration on it -- another example of special processing. (Granted, the filters are available form other vendors, but the basic fact that your film camera will not work with this film without special processing (the filters) remains.
As for another phase of the same problem, when you buy a given film, you may be required to buy a certain set of chemicals to process that film -- much equivalent to requiring special software to process a RAW format. (Except that with the digital cameras, you are not *forced* to use the RAW format -- the camera will happily provide you with JPEGs (the equivalent of carrying the film to the corner drugstore for processing.)
But you could not process Kodak Ektachrome with Agfa chemicals or Ansco chemicals. For that matter, you could not process Kodak's Kodachrome with Kodak's Ektachrome chemicals -- let alone processing Kodacolor.
Ektachrome even changed processes over time. Sometimes the new chemical set would work with the older film, and sometimes not. I know this, because I used to develop my own Ektachrome a *lot* -- from perhaps around 1962 on through about 1976, when I got married, and was working for a government organization which prohibited my bringing cameras on base with me, so my photo work pretty much stopped until I retired, and then the digital cameras came along.
And there was at least one very high resolution B&W film which required a special developer (was the maker H&W? I used it -- once.)
Granted -- for most B&W films, you could use a fairly wide range of chemicals from different vendors to develop it --- including mixing your own from old formulas -- or from knowing enough about the process to devise your own. (There was not enough information about the slide films to allow this freedom. And it is a lot harder to pick apart the photochemistry of a transparency film than an image format.)
But -- I don't see how the situation with RAW files is any worse than these limitations from the film days. And in some ways, it is a *lot* better. If you process a RAW file with the wrong software, the worst that will happen is that you won't get anything useful. The original RAW file can be saved until you get the *right* software.
Note that I prefer to do my image processing on unix computers, which are *not* supported by either Kodak or Cannon (or any of the other digital SLR makers, AFIK). For this reason, I am quite pleased to have Dave Coffin's "dcraw" software available.
Yes -- having all the vendors use a common format would be convenient (other than JPEG, which I detest as a lossy algorithm), but I'll expect that to happen about the time that all film maker's color processes are interchangeable. :-)
Enjoy, DoN.
 Signature Email: <dnichols@d-and-d.com> | Voice (all times): (703) 938-4564 (too) near Washington D.C. | http://www.d-and-d.com/dnichols/DoN.html --- Black Holes are where God is dividing by zero ---
John Francis - 27 May 2005 04:07 GMT > But -- I don't see how the situation with RAW files is any worse >than these limitations from the film days. In the film days, you could buy a different (relatively cheap) film to use in your (relatively expensive) camera. Nowadays the limitation is embedded in the expensive component, and you can't avoid it.
Barry Pearson - 27 May 2005 08:46 GMT [snip]
> But -- I don't see how the situation with RAW files is any worse > than these limitations from the film days. And in some ways, it is a > *lot* better. If you process a RAW file with the wrong software, the > worst that will happen is that you won't get anything useful. The > original RAW file can be saved until you get the *right* software. I think the analogy with film can be taken too far. For example, DNG could be more usefully compared with the developed negative, rather than the undeveloped negative. Then we would be talking about standardisation of film sizes to enable the negative to be processed "anywhere" and by lots of enlargers and scanning and printing equipment.
For asset management systems that handle many Raw formats, or decades in the future when we want to process old Raw files, it may be hard to "get the *right* software".
Sometimes we should just ask "what do we, as photographers or users of photographs, want?" We surely wouldn't want lots of variants of small negatives, nearly but not quite 35mm / 2 x 8 sprockets / 24mm x 36mm image size. That is what we have with Raw formats at the moment.
(I'll pass over the fact that the DNG logo has 2 x 7 sprockets, not 2 x 8!)
> Note that I prefer to do my image processing on unix computers, > which are *not* supported by either Kodak or Cannon (or any of the other > digital SLR makers, AFIK). For this reason, I am quite pleased to have > Dave Coffin's "dcraw" software available. [snip]
You've just made the case for OpenRAW , and even DNG. Dave Coffin's work is valuable, but it should never have been necessary. You should be able to obtain a variety of Unix-based converters, etc, which were based on proper specifications.
-- Barry Pearson http://www.barry.pearson.name/photography/ http://www.birdsandanimals.info/
Alan Browne - 27 May 2005 14:38 GMT > (I'll pass over the fact that the DNG logo has 2 x 7 sprockets, not 2 x > 8!) Cropped Digital Negative.
;-)
 Signature -- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm -- r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm -- [SI] gallery & rulz: http://www.pbase.com/shootin -- e-meil: Remove FreeLunch.
RichA - 27 May 2005 02:55 GMT >> > activity -- or even the media itself who regularly troll their audience >> > with prurient, irrelevant, or just plain false information. [quoted text clipped - 22 lines] >file formats are as open as can be, _despite_ the best efforts of >Nikon. Here we have an example of a prime case of the toadying "company man" not concerned with customers. It's a disease that afflicts many people who work in companies over 1000 people, or those who fantasize that their own work is more valuable than good customer service. -Rich
Jeremy Nixon - 27 May 2005 05:41 GMT > Reichman, et al, say they don't want "trade secrets" revealed, but they > are woefully clueless in that their demands for documentation of these > files is exactly that. That's nonsense. No one is asking Nikon for the process by which white balance is arrived at, only the conclusion. If that could lead to reverse engineering their process, then the fact that you can get it using Nikon's software would lead to the exact same eventuality.
 Signature Jeremy | jeremy@exit109.com
Owamanga - 26 May 2005 19:29 GMT >http://www.luminous-landscape.com/essays/raw-flaw.shtml Chicken liken twaddle.
"Aggh! the sky is falling!"
http://www.ongoing-tales.com/SERIALS/oldtime/FAIRYTALES/chicklicken.html
-- Owamanga! http://www.pbase.com/owamanga
Alan Browne - 26 May 2005 20:33 GMT >>http://www.luminous-landscape.com/essays/raw-flaw.shtml > > Chicken liken twaddle. Open RAW is best for everyone in the long term. There's no excuse for any data in the image created by the camera you own to be obfuscated or encrypted.
 Signature -- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm -- r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm -- [SI] gallery & rulz: http://www.pbase.com/shootin -- e-meil: Remove FreeLunch.
eawckyegcy@yahoo.com - 26 May 2005 21:14 GMT > Open RAW is best for everyone in the long term. 48V twisted pair POTS is a "standard" invented at the dawn of the telephone era. There are many engineers who wish we could rip the entire mess out of the ground, off the poles and start over again. Offer praises to Allah or whoever that cell phones, VOIP and the rest of it are doing the job indirectly.
Be very careful what you wish for.
Alan Browne - 26 May 2005 21:36 GMT >>Open RAW is best for everyone in the long term. > [quoted text clipped - 3 lines] > Offer praises to Allah or whoever that cell phones, VOIP and the rest > of it are doing the job indirectly. Backward compatibilty is not compromised by forward progress where data formats are concerned. Formats can get ever richer without s/w losing the ability to read old data correctly. That is if s/w and formats are designed and maintained correctly.
Or, if you prefer, open RAW does not mean 'cast in concrete'. Each OEM can do as he likes as long as the format is readable by all, and special data sections are clearly documented.
Regarding the olde telephone analog standard it has served extremely well for a very long time. Real engineers are too practical to "wish" for solutions that don't make economic sense. They adapt instead to growth without leaving anything of value behind. This backward compatibilty 'tax' is far cheaper than "ripping the entire mess out of the ground..."
 Signature -- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm -- r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm -- [SI] gallery & rulz: http://www.pbase.com/shootin -- e-meil: Remove FreeLunch.
eawckyegcy@yahoo.com - 26 May 2005 22:06 GMT >> 48V twisted pair POTS is a "standard" invented at the dawn of the >> telephone era. There are many engineers who wish we could rip the [quoted text clipped - 4 lines] > Backward compatibilty is not compromised by forward progress where data > formats are concerned. Then you must agree that OpenRAW is without a legitimate function. What's good for the goose, etc.
> Or, if you prefer, open RAW does not mean 'cast in concrete'. Each OEM > can do as he likes as long as the format is readable by all, and special > data sections are clearly documented. As I said, this will in many instances necessarily reveal trade secrets or other facets of the technology that the manufacturers would likely be unwilling to disclose. And if part of the OpenRAW is the signing of an NDA, then can we honestly call it "open"?
> Regarding the olde telephone analog standard it has served extremely > well for a very long time. Real engineers are too practical to "wish" > for solutions that don't make economic sense. If some djinn were to remove the need to work against the crazy POTS nonsense, many communications engineers would be profoundly thankful. Indeed, they must be: look at where all the innovation is occuring today re: telephony. I can cite similar software and hardware examples. _NO ONE_ likes to deal with legacy systems.
> This backward compatibilty 'tax' is far cheaper than "ripping the > entire mess out of the ground..." You are forgetting or ignoring the cost of innovations that are simply unimplementable within the legacy framework. It is for this and other reasons that the sensible person does not want camera manufacturers constrained by some Adobe or OpenRAW or otherwise committee drafted multi-volume 2356 page standard written in dense legalese.
Alan Browne - 26 May 2005 22:37 GMT >>>48V twisted pair POTS is a "standard" invented at the dawn of the >>>telephone era. There are many engineers who wish we could rip the [quoted text clipped - 7 lines] > Then you must agree that OpenRAW is without a legitimate function. > What's good for the goose, etc. I don't follow that one. What makes sense is that what gets recorded onto the memory card is the property of the photographer or employer, not the camera OEM.
>>Or, if you prefer, open RAW does not mean 'cast in concrete'. Each OEM >>can do as he likes as long as the format is readable by all, and special [quoted text clipped - 4 lines] > be unwilling to disclose. And if part of the OpenRAW is the signing of > an NDA, then can we honestly call it "open"? Since these trade secrets have a half life in the wild of about 3 days, they are no more useful to the OEM's than encryption has been for the DVD producers. NDA's don't protect anything of this nature.
"Trade Secret"? Why hasn't the dcraw author been sued into homelessness? If he can do it, the folks at competing OEM's can do it just as quick.
>>Regarding the olde telephone analog standard it has served extremely >>well for a very long time. Real engineers are too practical to "wish" >>for solutions that don't make economic sense. > > If some djinn were to remove the need to work against the crazy POTS > nonsense, many communications engineers would be profoundly thankful. But that's not engineering. The same could be said about many infrastructures from roadways to airways. If we were to design "World 2.0" do you think we would have LA, Mexico and Cairo (to name very few).
> Indeed, they must be: look at where all the innovation is occuring > today re: telephony. I can cite similar software and hardware > examples. _NO ONE_ likes to deal with legacy systems. If it is economically sensible to do so, it is done. That is engineering in large part. "Like" has little to with it.
>>This backward compatibilty 'tax' is far cheaper than "ripping the >>entire mess out of the ground..." [quoted text clipped - 4 lines] > constrained by some Adobe or OpenRAW or otherwise committee drafted > multi-volume 2356 page standard written in dense legalese. See above, re: World 2.0.
If that's so, why are they all committe members, participants and signatories to various other standards?
Feel free to have the last word ... this is it for me.
Cheers, Alan
 Signature -- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm -- r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm -- [SI] gallery & rulz: http://www.pbase.com/shootin -- e-meil: Remove FreeLunch.
Alan Browne - 26 May 2005 22:47 GMT > infrastructures from roadways to airways. If we were to design "World > 2.0" do you think we would have LA, Mexico and Cairo (to name very few). oops. Meant "Mexico City".
 Signature -- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm -- r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm -- [SI] gallery & rulz: http://www.pbase.com/shootin -- e-meil: Remove FreeLunch.
eawckyegcy@yahoo.com - 26 May 2005 23:20 GMT >>>Backward compatibilty is not compromised by forward progress where data >>>formats are concerned. [quoted text clipped - 3 lines] > > I don't follow that one. Hey, you made the claim. If your statement is true, then the camera makers are free to invent whatever new file formats they like ("forward progress") since "backward compatibility is not compromised". And with such freedom, what is OpenRAW's purpose?
> What makes sense is that what gets recorded > onto the memory card is the property of the photographer or employer, > not the camera OEM. Sounds like you need an IP attorney, not an OpenRAW standard.
>> As I said, this will in many instances necessarily reveal trade secrets >> or other facets of the technology that the manufacturers would likely [quoted text clipped - 4 lines] > they are no more useful to the OEM's than encryption has been for the > DVD producers. NDA's don't protect anything of this nature. Yes, yes, the manufacturers can make idiots of themselves. Why are you so concerned about this?
> "Trade Secret"? Why hasn't the dcraw author been sued into > homelessness? He hasn't published anything particularly sensitive yet? But even if he did, trade secrets, in the absence of a contract, have basically no legal protections at all.
> If he can do it, the folks at competing OEM's can do it > just as quick. Reverse engineering is an honoured tradition. What better compliment can you offer the manufacturer?
>> If some djinn were to remove the need to work against the crazy POTS >> nonsense, many communications engineers would be profoundly thankful. > > But that's not engineering. Of course it is. There are many situations, particularly in software, where one invents brand new stuff all the time. No legacy constraints. It is a most refreshing experience.
> The same could be said about many > infrastructures from roadways to airways. Your point?
> If we were to design "World > 2.0" do you think we would have LA, Mexico and Cairo (to name very few). I honestly couldn't say. And really, is this relevant?
>> Indeed, they must be: look at where all the innovation is occuring >> today re: telephony. I can cite similar software and hardware >> examples. _NO ONE_ likes to deal with legacy systems. > > If it is economically sensible to do so, it is done. That is > engineering in large part. "Like" has little to with it. "Like" has everything to do with it, since if nobody likes it, it won't get done. More importantly, though, if physical reality says "no", it doesn't matter what you think.
>>>This backward compatibilty 'tax' is far cheaper than "ripping the >>>entire mess out of the ground..." [quoted text clipped - 6 lines] > > See above, re: World 2.0. Ok, then, send Canon a check to cover their costs of abiding by the Official OpenRAW Standard, or to make up for lost revenue to products or services which they would like to produce, but which can not fit into this standard.
> If that's so, why are they all committe members, participants and > signatories to various other standards? Oh, I could rant about this at some length. But why should I entertain your distractions?
> Feel free to have the last word ... this is it for me. Thanks!
Barry Pearson - 27 May 2005 00:20 GMT [snip]
> Ok, then, send Canon a check to cover their costs of abiding by the > Official OpenRAW Standard, or to make up for lost revenue to products > or services which they would like to produce, but which can not fit > into this standard. [snip]
The cost of abiding by a common Raw format would probably be small. There would probably be a start-up cost - "climbing the learning curve". (If I were a manager in a camera-making company, I would prefer to manage the second camera in the company to use it, not the first!)
After that, it would probably be cheaper. There might be cost-savings such as common bits of firmware that could be bought, re-used, etc. And savings in learning about the format. There has been a lot of "wheel re-inventing" up to now, with must surely add to cost, and perhaps reduce quality.
No camera launched in the last year and a half would have needed a change to a well-engineered common Raw format. The vast majority of changes, such as more pixels, greater bit depth, etc, don't need changes. So there may not be any such products and services that they couldn't produce.
The one restriction that camera manufacturers would face is that they could not then use encryption to coerce their customers to buy their software.
-- Barry Pearson http://www.barry.pearson.name/photography/ http://www.birdsandanimals.info/
Barry Pearson - 27 May 2005 00:51 GMT [snip]
> As I said, this will in many instances necessarily reveal trade secrets > or other facets of the technology that the manufacturers would likely > be unwilling to disclose. And if part of the OpenRAW is the signing of > an NDA, then can we honestly call it "open"? Publicising current formats may reveal some things that manufacturers would not like revealed. Although that may be just the design quality of their Raw file formats! But interface specifications don't normally reveal much about the inside of the box. I would expect the formats to identify the sensor configuration, but the manufacturers are typically willing, sometimes eager, to tell the world about that anyway.
Use of a common Raw format would reveal even less. At the moment, the DNG format caters for Bayer sensors, Fujifilm SR sensors, Sony 4-colour sensors, and Foveon sensors. (Possibly lots more configurations could also be handled). If you can handle all of those with a single specification, then using that specification isn't revealing much about your technology. Formats designed specifically for your technology are more likely to be revealing, if only accidentally.
[snip]
> You are forgetting or ignoring the cost of innovations that are simply > unimplementable within the legacy framework. It is for this and other > reasons that the sensible person does not want camera manufacturers > constrained by some Adobe or OpenRAW or otherwise committee drafted > multi-volume 2356 page standard written in dense legalese. What innovations are those? No new cameras in the last year and a half would have needed a change to a common Raw format. The vast majority of innovations are not visible in details of the Raw file format.
The DNG specification is 50 pages long, and a lot of that is white space. It builds on TIFF/EP (the draft was 65 pages). But the camera manufacturers used that anyway! Typical proprietary Raw formats are already based on TIFF/EP. (And TIFF/EP refers to others). At the moment, internally the manufacturers must use these external specifications plus their own internal usage specifications. DNG, or other common Raw format, would replace some internal specification.
Any photographer concerned with the future health of top-end digital photography ought to want camera manufacturers to use a common Raw format. And DNG is suitable for that purpose.
-- Barry Pearson http://www.barry.pearson.name/photography/ http://www.birdsandanimals.info/
eawckyegcy@yahoo.com - 27 May 2005 01:24 GMT > Publicising current formats may reveal some things that manufacturers > would not like revealed. Although that may be just the design quality > of their Raw file formats! I can't comment on Nikon stuff, since I'm a Canon sort of person.
Canon 10D: a bizarre CIFF, for which they published the specifications for the "superstructure" of this file. It's a kind of weird TIFF; you can, in fact, TIFFize a CIFF. There are two images in this file: the JPEG and an strangely encoded raw sensor dump. That Dave Coffin managed to figure it out speaks for his abilities as a Reverse Engineer.
Canon 1DMkII: it's a straight TIFF, with EXIF markup. Three images embedded: a flat RGB thumbnail, a 1536 pixel wide JPEG (used for in-camera editing), and the lossless JPEG encoded sensor dump.
The main mysteries for both are not the bulk format, but the undocumented tags, specifically the "EXIF MakerNote" as well as the various non-standard TIFF tags.
> But interface specifications don't normally > reveal much about the inside of the box. I would expect the formats to > identify the sensor configuration, but the manufacturers are typically > willing, sometimes eager, to tell the world about that anyway. I have reverse engineered a fair amount of the 10D's undocumented tags (and some of the 1DMkII's -- too busy). There is one that does indeed lay out of the geometry of the sensor (how large, where the "image" starts, and so forth). There are other tags that detail the active AF sensor bitmap, various light meter readings, exposure computations and so forth. Just a few days ago, I found (by accident) the default luminance curve in the 1DMkII CR2 file.
Probably the most interesting aspect is one that is relevant here: the sensor has a 64 column bitmap "black" pixels on the left side of the image. The usual assumption is that these are used for bias level stuff. However, there is unusual stuff occuring here when the exposures grow to 30 seconds or more ... the 64 columns splits into distinct 16 column set and a 48 column set, and these columns are (apparently) used to produce a much better bias estimate than would be obtained from simple averaging.
This is precisely the sort of detail that Canon would almost certainly _not_ want to be generally known (it took me a while to figure it out), not just that it exists, but that someone might be able to figure out the device physics and go from there. If this is such a thing, though, there are almost certainly others awaiting.
Or maybe everyone knows this stuff because it is a Canon patented process.
> Use of a common Raw format would reveal even less. Suppose the above 16/48 column stuff was a Canon "trade secret" they would prefer to keep. How could a standard format encompass such sensor behaviour without revealing the secret? Or suppose that everyone knows this -- it is a Canon patent. Unless Canon licenses this patent for zero cost (which we can assume will be a zero-probability event), no one can use it even if it was fully documented to the last bit.
> Formats designed specifically for your technology are > more likely to be revealing, if only accidentally. I suggest this is more common than people think. The "MakerNote" information is tightly coupled to the hardware, at least it is for Canon equipment.
Jeremy Nixon - 27 May 2005 05:36 GMT > Suppose the above 16/48 column stuff was a Canon "trade secret" they > would prefer to keep. How could a standard format encompass such > sensor behaviour without revealing the secret? Does not the fact that DNG can *already* handle it indicate that this "problem" is nonexistent? Does not the fact that DNG can fully handle data from cameras that didn't exist at the time its code was written mean that this has no actual basis in reality?
 Signature Jeremy | jeremy@exit109.com
eawckyegcy@yahoo.com - 27 May 2005 17:06 GMT >> Suppose the above 16/48 column stuff was a Canon "trade secret" they >> would prefer to keep. How could a standard format encompass such >> sensor behaviour without revealing the secret? > > Does not the fact that DNG can *already* handle it indicate that this > "problem" is nonexistent? Cite the relevant sections in the DNG specification which detail exactly this -- that is, how to handle the 16/48 column stuff -- and you have made a point.
Jeremy Nixon - 27 May 2005 18:20 GMT > Cite the relevant sections in the DNG specification which detail > exactly this -- that is, how to handle the 16/48 column stuff -- and > you have made a point. It doesn't have to specify such a thing. That's not what DNG is, or what it does. It doesn't have to handle, or know about, every technical difference in every sensor design. Indeed, it proves that using a different RAW format for every camera model is completely unnecessary.
 Signature Jeremy | jeremy@exit109.com
eawckyegcy@yahoo.com - 27 May 2005 19:16 GMT >> Cite the relevant sections in the DNG specification which detail >> exactly this -- that is, how to handle the 16/48 column stuff -- and >> you have made a point. > > It doesn't have to specify such a thing. Yes it does. If the native camera format can do things that the DNG can't, why use the DNG?
> That's not what DNG is, or what it does. Shifting the goal-posts now? OpenRAW wants full, complete, documentation.
> It doesn't have to handle, or know about, every technical difference > in every sensor design. If it doesn't, then you lose capabilities that may be associated with the sensor. If the DNG removed the red channels from all RAW images, would you use it?
> Indeed, it proves that using a different RAW format > for every camera model is completely unnecessary. I have data that suggests that Canon cameras have unique black level estimation for long exposure images. The DNG format does not allow for such behaviour (or at least you have not presented me with a citation -- and I couldn't find one when the DNG spec was initially released).
Now if the DNG can't handle that, what do you recommend I do with my multi-minute minute exposure CR2 files I have?
Jeremy Nixon - 27 May 2005 19:32 GMT >> It doesn't have to specify such a thing. > > Yes it does. If the native camera format can do things that the DNG > can't, why use the DNG? It's not something the format, native or DNG, "does".
>> It doesn't have to handle, or know about, every technical difference >> in every sensor design. > > If it doesn't, then you lose capabilities that may be associated with > the sensor. No, you don't.
> I have data that suggests that Canon cameras have unique black level > estimation for long exposure images. The DNG format does not allow for [quoted text clipped - 3 lines] > Now if the DNG can't handle that, what do you recommend I do with my > multi-minute minute exposure CR2 files I have? It is not something DNG needs to "handle".
Try taking your files, converting them to DNG, and loading them into Camera Raw. It'll work.
 Signature Jeremy | jeremy@exit109.com
eawckyegcy@yahoo.com - 27 May 2005 19:44 GMT >> I have data that suggests that Canon cameras have unique black level >> estimation for long exposure images. The DNG format does not allow for [quoted text clipped - 5 lines] > > It is not something DNG needs to "handle". Then the DNG is useless re: the above situation, since the native CR2/CRW files (and their decoders from Canon) handle it just fine.
> Try taking your files, converting them to DNG, and loading them into > Camera Raw. It'll work. Something will happen, but probably not the _optimal_ thing. Maybe you have lesser goals, but I paid alot for my camera, and want my images given as good a treatment as possible. If DNG can't do it, why should I use it?
Jeremy Nixon - 27 May 2005 19:55 GMT >> It is not something DNG needs to "handle". > > Then the DNG is useless re: the above situation, since the native > CR2/CRW files (and their decoders from Canon) handle it just fine. You don't understand what DNG is. The native RAW files don't have to "handle" it, either. One has nothing to do with the other.
 Signature Jeremy | jeremy@exit109.com
eawckyegcy@yahoo.com - 27 May 2005 20:11 GMT Jeremy Nixon babbles:
> The native RAW files don't have to "handle" it, either. Once again, nitwit: I have data that hints that Canon has a special black-level estimator for long exposure images.
This estimator, if it exists, would be built into Canon CR2 file decoders.
None of this is known to DNG. (I am still awaiting a DNG spec citation.)
Thus no current DNG decoder uses this special estimator.
Given this context, why should I use DNG over CR2?
Oh, you might argue, but Canon could make a DNG decoder that knows all about that. Shall I respond to this now, or can you figure out the problem on your own?
Jeremy Nixon - 27 May 2005 20:41 GMT >> The native RAW files don't have to "handle" it, either. > > Once again, nitwit: I have data that hints that Canon has a special > black-level estimator for long exposure images. Good for you.
> This estimator, if it exists, would be built into Canon CR2 file > decoders. > > None of this is known to DNG. (I am still awaiting a DNG spec > citation.) Then I'm awaiting an explanation of why DNG should have to know about it. Given that it *already works* in DNG *right now*, I think it's pretty clear that it doesn't, any more than it has to know about every other camera that comes out.
I'm currently processing my camera's RAW files after converting to DNG. My camera didn't exist at the time the version of Camera Raw I'm using was written. Yet, somehow, it works perfectly. How do you suppose that is? How could DNG possibly "know about" my camera's unique RAW format?
Because it doesn't have to, of course.
> Given this context, why should I use DNG over CR2? You shouldn't. No one is trying to make you. But neither should you argue against DNG based on faulty understanding of what it is. You are assuming that something Canon is doing would require special consideration in the RAW file format; it does not require any such thing.
If something Canon is doing results in their RAW conversion software doing a better job than third-party software does, that is a completely different thing and it has nothing, at all, to do with the RAW file format.
 Signature Jeremy | jeremy@exit109.com
Barry Pearson - 27 May 2005 23:41 GMT [snip]
> I'm currently processing my camera's RAW files after converting to DNG. > My camera didn't exist at the time the version of Camera Raw I'm using > was written. Yet, somehow, it works perfectly. How do you suppose that > is? How could DNG possibly "know about" my camera's unique RAW format? > > Because it doesn't have to, of course. [snip]
Thomas Knoll told me that no camera of the last year and a half would have needed a change to DNG, had DNG existed that long ago.
Typically, to support a new camera, it isn't the DNG specification that needs to change. Instead, what has to happen is that something has to write, into specified places in the DNG file, specific information about the camera model. (In addition to the sensor data for the photograph, of course!) Then the Raw processing software, ACR or something else, can extract this specific information instead of having built-in knowledge of the camera.
I believe it works like this. Suppose I launch a new camera tomorrow, that writes Raw files with a BCP extension. I also release some software to process those Raw files. The camera has some existing sensor configuration, such as Bayer, or Fujifilm SR, or Sony 4-color, or Foveon, and writes just the sensor data to the BCP file. My software obviously knows what the sensor configuration is, and interprets the sensor data accordingly. This also includes interpreting the sensor values according to the colour temperature.
No other software has a clue how to interpret the sensor data. It won't even know what the sensor configuration is, and that is obviously vital, even before you worry about the colour temperature and other things. All Raw processing software will there have to give up on this BCP file format. Obviously, ACR 2.4 in Photoshop CS has to give up, because its built-in tables don't cover this camera.
Now Adobe releases a new version of the DNG Converter that knows about this camera. This DNG Converter writes the sensor data to the DNG file, of course. And also writes fields that describe the sensor configuration, and how to interpret the colour temperature, etc. So now the DNG file is "self contained". Some fields provide the sensor data for the photograph. Others identify how to interpret that data. (Metadata).
Now, when ACR 2.4, or any other Raw processor that handles DNG, reads that file, it doesn't need its own data to tell it how to interpret the data for this camera. It just extracts that from the DNG file. It can handle cameras it has never before come across, because the DNG file itself tells it how to. (I think the DNG Converter writes colour calibration fields for 2 colour tempertaures, one tungsten and the other daylight, to the file. This enables ACR to do its white balance handling).
I'm still hazy about some of the details. But what I've said above is consistent with everything I've learned and tested so far. I have converted D2X and 350D Raw files into DNG, and processed them in ACR 2.4 under CS. CS Browser shows thumbnails for the DNG files just as it would any Raw files that it recognised, and ACR 2.4 can process them. But it just shows place-holders for the original Raw files, and refuses to open the latter because it doesn't recognise the format.
It is this concept of being "self contained", with the sensor data for the image, plus metadata identifying how to interpret the sensor data, that makes DNG very powerful. Someone could release a photo-editor tomorrow that only accepted DNG, and it would be able to process all 77 cameras that the DNG Converter could handle. Someone could release a new camera tomorrow that recorded DNG, and immediately all Raw processors that handled DNG properly would support that camera. And in decades to come, DNG processors will be able to handle old DNG files that have images taken using cameras that no one knows about anymore.
-- Barry Pearson http://www.barry.pearson.name/photography/ http://www.birdsandanimals.info/
Jeremy Nixon - 28 May 2005 01:32 GMT > The camera has some existing sensor configuration, such as Bayer, or > Fujifilm SR, or Sony 4-color, Indeed, I don't think the Sony 4-color requires any special consideration in DNG, as per some statements I've seen from Thomas Knoll. It's still a Bayer pattern, and the particular colors in use apparently don't matter since they are specified in the DNG file.
> I'm still hazy about some of the details. But what I've said above is > consistent with everything I've learned and tested so far. I have [quoted text clipped - 3 lines] > But it just shows place-holders for the original Raw files, and refuses > to open the latter because it doesn't recognise the format. I've used it for my D2x files, too, in ACR 2.4, with complete success. It still does a better job than either dcraw or Nikon's own software, despite having been written before the D2x existed.
 Signature Jeremy | jeremy@exit109.com
eawckyegcy@yahoo.com - 28 May 2005 00:27 GMT >> Once again, nitwit: I have data that hints that Canon has a special >> black-level estimator for long exposure images. > > Good for you. I know, it is. What data do you have, homeboy?
> > This estimator, if it exists, would be built into Canon CR2 file > > decoders. [quoted text clipped - 3 lines] > > Then I'm awaiting an explanation of why DNG should have to know about it. Ohhhhh, I dunno, in order to make a better black level estimate?
> Given that it *already works* in DNG *right now*, I think it's pretty > clear that it doesn't, any more than it has to know about every other > camera that comes out. Well, maybe your code is making non-optimal estimates. Oh my. Oh me.
> I'm currently processing my camera's RAW files after converting to DNG. > My camera didn't exist at the time the version of Camera Raw I'm using > was written. Yet, somehow, it works perfectly. How do you suppose that > is? How could DNG possibly "know about" my camera's unique RAW format? > > Because it doesn't have to, of course. Your converter can make as many assumptions as it likes, of course. Maybe you like guessing: I like knowing.
> > Given this context, why should I use DNG over CR2? > > You shouldn't. No one is trying to make you. But neither should you > argue against DNG based on faulty understanding of what it is. Take your straw-men and shove 'em. Now answer the question as posed: if Canon software can render a better image than a DNG decoder, why use the DNG decoder?
> You > are assuming that something Canon is doing would require special > consideration in the RAW file format; it does not require any such > thing. I am not assuming Canon is doing something: _I KNOW THEY ARE_. Remember the "data" that is "good for me"?
> If something Canon is doing results in their RAW conversion software > doing a better job than third-party software does, that is a completely > different thing and it has nothing, at all, to do with the RAW file format. Canon's converters will do a better job becaues they have knowledge of (a) their hardware, but more importantly, (b) specific information in their CR2 files which directs superior handling of the raw image data. SHOCK OF SHOCKS: some of this information may not be encodable in the DNG file!
Jeremy Nixon - 28 May 2005 01:27 GMT >> Then I'm awaiting an explanation of why DNG should have to know about it. > > Ohhhhh, I dunno, in order to make a better black level estimate? DNG does not make a black level estimate at all, so making a better one is really not relevant.
The information you're using to make that black level estimate can be preserved in a DNG file, though, so that software that knows how to use it will be able to.
> Take your straw-men and shove 'em. Now answer the question as posed: > if Canon software can render a better image than a DNG decoder, why use > the DNG decoder? Absolutely no reason at all. None whatsoever. No one is trying to say that you should do that. Indeed, if you prefer the results you get with Canon's software, I would say that you should use it.
> I am not assuming Canon is doing something: _I KNOW THEY ARE_. > Remember the "data" that is "good for me"? You know Canon is doing something; you assume that it will require specific support in the RAW file format, and that assumption is not correct.
> Canon's converters will do a better job becaues they have knowledge of > (a) their hardware, but more importantly, (b) specific information in > their CR2 files which directs superior handling of the raw image data. If Canon's converters do a better job, then go ahead and use them.
> SHOCK OF SHOCKS: some of this information may not be encodable in the > DNG file! Except that it is, so that doesn't apply. It *is* possible to do something that can't be encoded by the existing DNG specification, but Canon hasn't done so, nor has any camera since (I believe) the Fuji 2-level sensor thing.
 Signature Jeremy | jeremy@exit109.com
JPS@no.komm - 28 May 2005 02:02 GMT >Then I'm awaiting an explanation of why DNG should have to know about it. >Given that it *already works* in DNG *right now*, What do you mean by "already works"? How do you know the conversion couldn't be better? DNG does not keep all black pixels; it has a tag that *may* report the difference for each horizontal and vertical black line from the global blackpoint it writes to the file.
 Signature
<>>< ><<> ><<> <>>< ><<> <>>< <>>< ><<> John P Sheehy <JPS@no.komm>
><<> <>>< <>>< ><<> <>>< ><<> ><<> <>>< JPS@no.komm - 28 May 2005 01:59 GMT >Jeremy Nixon babbles: > [quoted text clipped - 10 lines] > >Thus no current DNG decoder uses this special estimator. It is possible that the DNG ENcoder takes it into account, though.
That's the way DNG 3.1 deals with 20D banding, I think; when it writes the DNG, it alters the RAW data so that it needs the same blackpoint for every line.
>Given this context, why should I use DNG over CR2? > >Oh, you might argue, but Canon could make a DNG decoder that knows all >about that. Shall I respond to this now, or can you figure out the >problem on your own? Or, Canon might allow for the camera to record this information, but Canon never actually uses it; just like they don't use the individual lines' black data in their RAW processors, but DNG 3.1 *does*.
 Signature
<>>< ><<> ><<> <>>< ><<> <>>< <>>< ><<> John P Sheehy <JPS@no.komm>
><<> <>>< <>>< ><<> <>>< ><<> ><<> <>>< Owamanga - 27 May 2005 20:15 GMT >>> It is not something DNG needs to "handle". >> [quoted text clipped - 3 lines] >You don't understand what DNG is. The native RAW files don't have to >"handle" it, either. One has nothing to do with the other. Then explain, because your comments have me confused now.
Either DNG is:
1) Simply a container, imbecilically storing the RAW file within without any comprehension as to its content. Basically no better than zipping the damn RAW file up.
2) A new representation of the data within the RAW file, which therefore must either understand and re-represent every aspect of the original RAW file in an open format manner or be considered a lossy storage method.
3) Some hybrid of the above, meaning it must be significantly larger than the RAW file originally was.
Until DNG is so good, that in every case, once a RAW is converted to DNG we can safely throw the original RAW file away and have lost nothing, it'll have extremely limited appeal.
Even if Adobe's solution promises to be able to recreate the original RAW file from the DNG, that's really zero steps better than a zip file.
-- Owamanga! http://www.pbase.com/owamanga
Barry Pearson - 27 May 2005 21:54 GMT [snip]
> Either DNG is: > > 1) Simply a container, imbecilically storing the RAW file within > without any comprehension as to its content. Basically no better than > zipping the damn RAW file up. No. Code can be written that uses the DNG specification to turn a DNG-conformant file into a recognisable image, without needing to know anything about the camera concerned. So DNG obviously has the necessary semantics.
(DNG is a variant of TIFF/EP. So is the typical Raw file format. Typically, much of the important data has the same meaning in both formats).
> 2) A new representation of the data within the RAW file, which > therefore must either understand and re-represent every aspect of the > original RAW file in an open format manner or be considered a lossy > storage method. No. The essential image data is understand, but often not re-represented. Some contents of Raw file formats are already likely to conform to DNG, because they conform the recognised parts of TIFF/EP. If a Raw format already has the sensor data in the required format, why re-represent it? (The fact that a file format is proprietary and not open doesn't mean that "every aspect of the original RAW file" is closed, not open. Surely the bits that conform to TIFF/EP can be considered to be open?).
In addition, DNG offers the option of including non-essential private data that is not dictated by the specification. So it doesn't understand "every aspect".
> 3) Some hybrid of the above, meaning it must be significantly larger > than the RAW file originally was. No. In all cases that I've tested, without embeding the original file, the DNG version was smaller than the original version. But only when losslessly compressed. So this is probably just a visible indication of the relative effectiveness of the lossless compression used by DNG compared with the compression used in the original format.
(Note that even a D2X NEF file saved by Nikon Capture 4.2.1, while smaller than that output by the D2X camera, is larger than the DNG version. I think DNG simply uses a very effective lossless compression scheme).
> Until DNG is so good, that in every case, once a RAW is converted to > DNG we can safely throw the original RAW file away and have lost > nothing, it'll have extremely limited appeal. No. I don't save the original after converting to DNG. I've been losing things converting to DNG, such as the lens-model used for the shot. But those things haven't been important enough to me to do otherwise. Lots of other people appear to think likewise.
My guess is that DNG will be the format of choice for most Raw shooters who use CS2 and ACR 3.1. It is so convenient. And with about 25 non-Adobe products supporting DNG, its use won't be confined to Photoshop / ACR.
> Even if Adobe's solution promises to be able to recreate the original > RAW file from the DNG, that's really zero steps better than a zip > file. No. One method of being able to recreate the original Raw file from the DNG is to embed the original file in the DNG. Apart from the size problem, this can be attractive. The file behaves, of course, just like a DNG file, and can therefore be used in all cases where DNG would be used. But the original can be extracted with bit-level accuracy at any time.
Frankly, I haven't a clue whether the "unique black level estimation for long exposure images" is catered for by the current version of DNG. Since I don't use Canon, I have no need to think about it. Why couldn't it be held in DNGPrivateData, or in MakerNote with MakerNoteSafety set to 1 (safe)? If it can't, then Canon could ask for a new version. That is why DNG has a good version scheme.
-- Barry Pearson http://www.barry.pearson.name/photography/ http://www.birdsandanimals.info/
eawckyegcy@yahoo.com - 27 May 2005 23:01 GMT > Frankly, I haven't a clue whether the "unique black level estimation > for long exposure images" is catered for by the current version of DNG. It isn't.
> Since I don't use Canon, I have no need to think about it. www.google.com: define:abstract
Nikon (or whoever) has likely has something that is being dropped on the floor when you convert to DNG. This is the basic point (that is apparently lost on too many people): without specific support for all features present in a camera, DNG is necessarily lossy.
> Why couldn't > it be held in DNGPrivateData, or in MakerNote with MakerNoteSafety set > to 1 (safe)? Because no standard DNG reader will know what to do with it, why bother?
> If it can't, then Canon could ask for a new version. Why should they? Instead they can make a special reader which does know what to do with the "PrivateData" ... but hey, wait a minute ...
> That is why DNG has a good version scheme. The existence of "PrivateData" is the problem, not the lack or presence of version numbering.
In short:
a) if no DNG standard reader can understand the "private" stuff, why bother storing it at all? In this case, DNG offers you nothing more than what a TIFF file would.
b) if some DNG reader _can_ understand it, it must be a custom reader, and thus one has just converted the "file format problem" into an "application format problem". If you have to keep track of reader X to best deal with DNG's that come from camera X in order to get the most out of the images, in what way would this be different from the current situation? In this case, DNG has given you nothing but an extra, lazy, man in the middle that drops bits on the floor.
Note that in both cases the DNG (or OpenRAW or whatever) gives you nothing. Why, then, use it?
Jeremy Nixon - 27 May 2005 23:18 GMT > Nikon (or whoever) has likely has something that is being dropped on > the floor when you convert to DNG. This is the basic point (that is > apparently lost on too many people): without specific support for all > features present in a camera, DNG is necessarily lossy. Except that this is not correct. DNG does *not* need specific support for all camera features in order to preserve them; indeed, that's part of the whole point of the thing.
 Signature Jeremy | jeremy@exit109.com
eawckyegcy@yahoo.com - 28 May 2005 00:10 GMT >> Nikon (or whoever) has likely has something that is being dropped on >> the floor when you convert to DNG. This is the basic point (that is [quoted text clipped - 4 lines] > for all camera features in order to preserve them; indeed, that's part > of the whole point of the thing. Even taken alone, your statement is nonsensical. It is isomorphic to "You don't need an arm in order to use an arm."
Start making sense, dude.
Barry Pearson - 28 May 2005 08:58 GMT [snip]
> > Except that this is not correct. DNG does *not* need specific support > > for all camera features in order to preserve them; indeed, that's part > > of the whole point of the thing. > > Even taken alone, your statement is nonsensical. It is isomorphic to > "You don't need an arm in order to use an arm." [snip]
Let's step back to see what is being talked of here:
DNG is a specification of a file format. It can cater for a whole range of sensor configurations, including Bayer-like, Foveon-like, Fujifilm-like, Sony-like, and others. It has various parameters that provide extra details for each of these.
Any particular DNG image file uses just the subset which this camera needs, out of the vast range of possibilities. So that file is doing 2 important things, as far as this discussion is concerned. It identifies those details of the camera that will be needed when processing the image. And it supplies the image data itself.
It isn't important whether the DNG specification took this particular camera model into account. What matters is that the specification has sufficient richness to cater for this camera model. So a new camera model that differs only in bits-per-pixel, or number of X and Y pixels, or different look-up tables to transform the sensor data according to colour temperature, can be catered for.
It appears to be the case that DNG specification version 1.1.0.0, released in January, had sufficient richness for the 350D and D2X details. In fact, probably version 1.0.0.0, released last September, already had sufficient richness.
It is beyond doubt that ACR 2.4, released in January, already had the code within it to read DNG files generated from D2X NEFs and 350D CR2s, and process those images. I've tried this route, and I guess that by now lots of other people have. I wonder whether ACR 2.3, released in September, also had sufficient code within it to handle those DNG files? I suspect it did, because there are only small differences between versions 1.0.0.0 and 1.1.0.0.
Yet neither ACR 2.3 nor ACR 2.4 can read a D2X NEF file or a 350D CR2 file. It takes specific knowledge of a camera model to process Raw files, and ACR 2.x didn't have that knowledge. (Neither, in fact, did ACR 3.0).
This shows the power of DNG. You can write code to process files conforming to particular versions of DNG, and you can cater automatically for cameras that you haven't heard of, and that may not have existed when you wrote the code. The advantage for the camera manufacturer would be that their new camera is immediately supported by all Raw processors that catered properly for the chosen version of DNG. If a future D3X or 375D output in DNG version 1.1.0.0, then ACR 2.4 could process the images.
When new technology cannot be catered for by the current DNG version, a new version, for example 2.0.0.0, will be needed. That presumably can't be processed by ACR 2.4, nor other Raw processors. Upgrades will be needed.
-- Barry Pearson http://www.barry.pearson.name/photography/ http://www.birdsandanimals.info/
McLeod - 28 May 2005 15:51 GMT >This shows the power of DNG. You can write code to process files >conforming to particular versions of DNG, and you can cater [quoted text clipped - 9 lines] >be processed by ACR 2.4, nor other Raw processors. Upgrades will be >needed. What I would like to see is more open source image editing systems.
Adobe Systems is making me very nervous with their attempt to commoditze digital imagery. Once they have got us all using their image processing system what is to stop them from hitting us at every turn for more money. (Like CS2 and ACR 3.1).
From the Dave Coffin interview at dpreview http://www.joelonsoftware.com/articles/StrategyLetterV.html
Barry Pearson - 28 May 2005 17:44 GMT [snip]
> What I would like to see is more open source image editing systems. So would I. I am especially interested in John Francis' "Raw-to-DNG converter" SourceForge project, which will remove the one case where we are currently dependent on a single supplier. I am looking forward to there being a Converter that isn't constrained by Adobe's legal issues.
> Adobe Systems is making me very nervous with their attempt to > commoditze digital imagery. Once they have got us all using their > image processing system what is to stop them from hitting us at every > turn for more money. (Like CS2 and ACR 3.1). [snip]
Photoshop (and other photo-editors) have been tending to commoditise digital imagery since TIFF 6.0! It is nothing new. Everything that can deliver a TIFF 6.0 file into Photoshop is having to compete with everything else that can do it. But it can argued that, by outputting in TIFF 6.0 format, the camera manufacturers have been tending to commoditise photo-editors that accept TIFF 6.0 files, by forcing them to compete with one another. It isn't just a one-sided activity!
Now Adobe Camera Raw (and other Raw processors) are tending to commoditise products that output in Raw format. The latter are all having to compete with one another. And, conversely, the camera manufacturers are tending to commoditise Raw processors, of course.
But what Adobe are doing with DNG is very different from "Photoshop versus camera manufacturers". Adobe are trying to open up the whole industry and marketplace of Raw shooting and processing. Paradoxically, opening up an industry starts to remove the "dominant supplier" bias. If/when DNG becomes ubiquitous, people will be able to choose their Raw processors and asset management systems purely on merit. And they will also be able to choose their cameras purely on merit, instead of having to be guided by the fact that the main camera manufacturers tend to own the better software, and are better supported by Raw processors and asset management systems.
In what way are Adobe "hitting us at every turn for more money. (Like CS2 and ACR 3.1)"? If you don't want to upgrade, don't! Why will you ever need to pay Adobe any more money? As shown here, if you buy a camera that CS and ACR 2.4 doesn't support, use the free DNG Converter to convert that camera's Raw files to DNG, and you will be able to stick with CS and ACR 2.4. That route *really* does work! I've tried it, and so have many others.
If/when DNG becomes ubiquitous, we will all be able to build our digital photography system by choosing individual components, such as cameras, Raw processors, photo-editors, etc, on their merits. We will be able to do mix-n-match. Perhaps we will sometimes have alternatives to use for different purposes. This is the sort of future for top-end digital photography that all photographers, and users of photographs, should be hoping for.
What will Adobe get out of all this? Simple! The more freedom people have to choose, the more likely it is that a large proportion of them will buy Adobe products. Because they are the highest value digital imaging products around. THAT is why people will buy Photoshop; choice, not force.
-- Barry Pearson http://www.barry.pearson.name/photography/ http://www.birdsandanimals.info/
Barry Pearson - 28 May 2005 09:58 GMT > > Frankly, I haven't a clue whether the "unique black level estimation > > for long exposure images" is catered for by the current version of DNG. > > It isn't. What evidence is there for this?
[snip]
> Nikon (or whoever) has likely has something that is being dropped on > the floor when you convert to DNG. This is the basic point (that is > apparently lost on too many people): without specific support for all > features present in a camera, DNG is necessarily lossy. I've just covered this in another response. What matters is whether DNG has sufficient richness for all the details of the camera model needed for processing the Raw file. It appears to me that it has for the D2X and the 350D, because I could get good images via this route. But I accept that this was just for a limited number of images. I didn't process any that had a long exposure.
Had I tried such images, what problems would I have had? Is it something that couldn't be handled even in theory by Raw processors that didn't read the "unique black level estimation for long exposure images"? Or simply something that has to be catered for another way?
Remember the fuss about D2X WB encryption. DNG files generated from D2X Raw files don't (currently) properly identify the "as shot" WB, so in that sense those *files* are lossy. And for a lot of photographers, this turns out to be unimportant. But it doesn't mean that the DNG *format* is inherently lossy; what is lossy is the conversion process when performed by software that can't get at the WB. If the D2X camera were to output DNG files, it might well choose to do the job properly, and avoid loss. An open source DNG Converter, unconstrained by Adobe's fears of legal action, might write more complete DNG files from the D2X. I think we will see some interesting open source, and other independent, applications of DNG over the next year or so.
[snip]
> The existence of "PrivateData" is the problem, not the lack or presence > of version numbering. [quoted text clipped - 4 lines] > bother storing it at all? In this case, DNG offers you nothing more > than what a TIFF file would. Not true. For example, DNGs generated from the Raw files of my camera have data that isn't understood by standard DNG readers, and it makes no significant difference to me. I can't get at the lens model. If I could, it might be possible to write a script to analyse all the lens information in the metadata and apply automatic chomatic abberation reduction. But, in the absence of that, I do it by eye.
I think this is the likely use of the private data. To give the camera manufacturer's software opportunities to do things an easy way that other processors have to do a harder way. (Without offering such a "unfair advantage" to the camera manufacturers, it would be far harder to persuade them to use DNG. Perhaps it is a "necessary evil").
DNG gives me, and others who use it, the advantages of Raw processing over TIFF processing.
> b) if some DNG reader _can_ understand it, it must be a custom reader, > and thus one has just converted the "file format problem" into an [quoted text clipped - 3 lines] > situation? In this case, DNG has given you nothing but an extra, lazy, > man in the middle that drops bits on the floor. Not true. See my example above. I agree that a formal standard for archival purposes would be unlikely to have private data fields. Proposals for an archival version of PDF, called PDF-A, don't have encryption, for example. I can envisage an archival version of DNG, perhaps DNG-A, that has no private bits.
> Note that in both cases the DNG (or OpenRAW or whatever) gives you > nothing. Why, then, use it? I have been using DNG since 10th October 2004. It gives me benefits, more with each version of ACR, more with each version of the DNG Converter, and lots more with the upgrade to CS2.
I get much smaller files, which makes processing faster and archiving easier. I get my ACR settings and adjustments stored in the DNG file, so that I have just one file to deal with. I have the confidence that I will be able to process those archived DNG files for as long ahead as I need, and I didn't have this confidence with the files directly from my camera.
I know that ACR gives the same image whether I process the camera's original files or DNG files. (I have tested Raw files for a number of cameras of a number of makes, and in each case the result of the ACR processing was the same for both the original Raw file and the DNG version). If I buy a new camera after CS3 and ACR 4.x have been released, I will probably not have to upgrade to CS3 to handle that new camera.
The 3.1 DNG Converter stores the private data of my camera in the DNG file. Although I can't currently do anything with it, at least it is there. Perhaps in future I will be able to do something with it. Perhaps a new DNG Converter will accept my current DNG files and move that data to a place where it can be used directly. Who knows?
I think in future we will see high quality magazines, and other users of photographs, accepting Raw files. I'm confident they will accept DNG. They may sometimes insist upon it, just as now they may insist upon TIFF 6.0.
DNG is opening up the world of Raw processing. I think we will see benefits in future that we can't yet think of.
-- Barry Pearson http://www.barry.pearson.name/photography/ http://www.birdsandanimals.info/
Owamanga - 27 May 2005 19:35 GMT >Now if the DNG can't handle that, what do you recommend I do with my >multi-minute minute exposure CR2 files I have? Dunno, but based on the original article, I'm just looking forward to playing all my Betamax tapes on my VHS recorder again, which it almost promises the DNG standard will be capable of doing.
Traci Lords anyone?
-- Owamanga! http://www.pbase.com/owamanga
Barry Pearson - 27 May 2005 08:29 GMT [snip]
> I can't comment on Nikon stuff, since I'm a Canon sort of person. > [quoted text clipped - 4 lines] > managed to figure it out speaks for his abilities as a Reverse > Engineer. [snip]
By "weird TIFF", do you mean "weird TIFF 6.0" or "weird TIFF/EP"? Is it, in fact, "standard TIFF/EP"? You may know the following, but I suspect many people don't:
1. *TIFF 6.0* is owned by Adobe, and was owned by Aldus before that.
When you set your camera to TIFF, that really means TIFF 6.0. Raw processors normally have TIFF 6.0 as an *output* option. Photo-editors normally read and/or write TIFF 6.0 as options. (You can actually use TIFF 6.0 instead of PSD for Photoshop - make sure you ask for layers to be preserved). If a magazine buys one of your photographs in TIFF form, they mean TIFF 6.0.
TIFF 6.0 images have the full colour information for each pixel, (except for monochrome images, of course). TIFF 6.0 is not suitable for Raw files, because it has no tags defined for holding sensor data, and there are many other metadata tags that it would need also.
2. *TIFF/EP* ("Electronic Photography") is owned by ISO, and costs 150 euros. It was developed from TIFF 6.0, and much of it is copied directly from TIFF 6.0.
It has those extra tags for sensor data, (look for tags starting CFA - Color Filter Array). It also has a lot of extra metadata tags. Camera manufacturers typically (but not necessarily) start with TIFF/EP when designing their Raw formats, because it has many of the tags they need. Indeed, one manufacturer originally used the .TIF extension for its Raw files.
Like TIFF 6.0, TIFF/EP has very many options. And, of course, although it is a standard, there is no constraint on manufacturers to conform precisely to it. So resultant Raw formats proliferate, and there are lots of unnecessary differences.
3. *DNG*, like TIFF/EP, is a development of TIFF 6.0, but I prefer to think of it as a development of TIFF/EP. It has a few tags that TIFF/EP doesn't have, to cater for technology innovations since TIFF/EP was defined, and for control purposes, such as the 2 version tags that define the range of DNG versions that a particular file is compatible with.
DNG cuts through all the unnecessary options in TIFF/EP, eliminates some of the more complicated structural options, and makes some tags mandatory. So it ensures that the image data can always be extracted, along with sufficient extra information that Raw processors can handle the image data without needing to know about the camera. (The file is self-contained). But it also defines how camera manufacturers can safely store any "secret sauce" in a DNGPrivateData tag defined for the purpose. "Safely" means that anyone else can re-write a DNG file while preserving the private data, because although its contents are private, its format is well-structured.
4. To summarise: TIFF 6.0 is what people tend to mean when they say "TIFF" without qualification. TIFF/EP is TIFF 6.0 plus a lot of the stuff needed to make Raw files. DNG is TIFF/EP brought up to date and made fit for purpose. (Now I'll duck for cover!)
-- Barry Pearson http://www.barry.pearson.name/photography/ http://www.birdsandanimals.info/
eawckyegcy@yahoo.com - 27 May 2005 17:16 GMT > By "weird TIFF", do you mean "weird TIFF 6.0" or "weird TIFF/EP"? I mean "weird TIFF", as "unusual", "not exactly the same as but similar to in strange way", or "derived from by those under the influence of unspecified narcotic agents", and so forth. It has a directories, tags, and the rest of it. Everything else is completely incompatible with TIFF as is, but it can be easily converted into whatever TIFF one wishes (if anyone was so inclined).
> But it also defines how camera manufacturers can safely store any "secret > sauce" in a DNGPrivateData tag defined for the purpose. "Safely" means > that anyone else can re-write a DNG file while preserving the private > data, because although its contents are private, its format is > well-structured. You need to read the ranting of the OpenRAW people. They explicitly demand "no secret sauce" -- though irrationally maintaining they are not making such a demand.
Ben Rosengart - 27 May 2005 19:35 GMT > 4. To summarise: TIFF 6.0 is what people tend to mean when they say > "TIFF" without qualification. TIFF/EP is TIFF 6.0 plus a lot of the > stuff needed to make Raw files. DNG is TIFF/EP brought up to date and > made fit for purpose. (Now I'll duck for cover!) Thanks for this explanation. I found it quite enlightening.
 Signature Ben Rosengart (212) 741-4400 x215 Sometimes it only makes sense to focus our attention on those questions that are equal parts trivial and intriguing. --Josh Micah Marshall
Ryadia@Home - 26 May 2005 21:36 GMT >>Open RAW is best for everyone in the long term. > > Be very careful what you wish for. Yes. Mrs Browne wished for a boy with good looks, smart wit and high intellect. Unfortunately she got Alan!.
Douglas
Alan Browne - 26 May 2005 21:45 GMT > Yes. Mrs Browne wished for a boy with good looks, smart wit and high > intellect. Unfortunately she got Alan!. Your mother thought anal sex with a donkey wouldn't get her pregnant.
 Signature -- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm -- r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm -- [SI] gallery & rulz: http://www.pbase.com/shootin -- e-meil: Remove FreeLunch.
Barry Pearson - 27 May 2005 01:02 GMT > > Open RAW is best for everyone in the long term. > [quoted text clipped - 3 lines] > Offer praises to Allah or whoever that cell phones, VOIP and the rest > of it are doing the job indirectly. Perhaps, without that standard, we would not be advanced enough to make that wish! We would still be wasting too much time trying to get basic communications to work.
We sort out one layer of the communications interworking. That enables us to build lots of services in higher layers. That creates a desire for better lower layers. The trick is to have an architecture, or structural decomposition, that enable to make progress in one area without having to tear up all other areas. And that needs clean interfaces between the various areas.
If we DO decide to rip all that stuff out, I would hope we could do so without (say) having to re-invent TCP/IP to run over its successor.
> Be very careful what you wish for. I would think that anyone who has spent many years or decades helping to design complex multi-vendor systems knows what to wish for here! We need a common Raw format. And in the meantime we need sufficient publication of our current formats to build stop-gap measures such as access to proprietary Raw formats, and conversion to a common Raw format, etc.
-- Barry Pearson http://www.barry.pearson.name/photography/ http://www.birdsandanimals.info/
Owamanga - 26 May 2005 22:55 GMT >>http://www.luminous-landscape.com/essays/raw-flaw.shtml >> [quoted text clipped - 3 lines] >any data in the image created by the camera you own to be obfuscated or >encrypted. With that I agree, my complaint is with stupidity such as this:
"What happens when your new Quatum Cube based computer no longer can read CDs, or DVDs, or its operating system can't deal with something as old and arcane as Windows XP or Mac OSX?
Far fetched you say? We'll, how many of you have a stack of 3.5" floppies somewhere in your des
|
|