Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
PhotoKB Home
Discussion Groups
Digital Photography
Digital PhotoDSLR CamerasZLR CamerasPoint & Shoot Cameras
Film Photography
35 mmLarge FormatMedium formatDarkroomFilm and LabsOther Equipment
Photo Technique
Nature PhotographyPeople PhotographyTechnique General
General Photo Topics
General TopicsAustralian PhotographyUK Photography
DirectoryPhoto Clubs

Photo Forum / Digital Photography / DSLR Cameras / May 2005

Tip: Looking for answers? Try searching our database.

"Raw" file issues?

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
RichA - 26 May 2005 17:31 GMT
http://www.luminous-landscape.com/essays/raw-flaw.shtml
eawckyegcy@yahoo.com - 26 May 2005 19:07 GMT
RichA trolls:

> http://www.luminous-landscape.com/essays/raw-flaw.shtml

http://en.wikipedia.org/wiki/Internet_troll

Some day I will expand on this entry, by noting that "trolling"
behaviours are hardly limited to the Internet -- computer networks did
not magically allow people to suddenly make straw-man arguments.
Consider the classic example of a politician -- our polite name for
those who fabricate false arguments/statements as a professional
activity -- or even the media itself who regularly troll their audience
with prurient, irrelevant, or just plain false information.
Alan Browne - 26 May 2005 20:29 GMT
> activity -- or even the media itself who regularly troll their audience
> with prurient, irrelevant, or just plain false information.

Don't be an a.s.  Given the indications that companies like Nikon would
like nothing better than to sell more s/w as aprt of their camera
solution (other OEM's too), everyone has everything to gain by a
standard for RAW images.

Signature

-- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm
--        r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm
--      [SI] gallery & rulz: http://www.pbase.com/shootin
--                   e-meil: Remove FreeLunch.

eawckyegcy@yahoo.com - 26 May 2005 21:10 GMT
> > activity -- or even the media itself who regularly troll their audience
> > with prurient, irrelevant, or just plain false information.
[quoted text clipped - 3 lines]
> solution (other OEM's too), everyone has everything to gain by a
> standard for RAW images.

It is not in Nikon's interest to tell everyone their innovations in
(say) automatic white balance, or in Canon's interest to spill their
beans re: (say) fancy device physics for optimal bias estimation in
long exposure images.

Reichman, et al, say they don't want "trade secrets" revealed, but they
are woefully clueless in that their demands for documentation of these
files is exactly that.

If you don't like Nikon's or Canon's policies about any of this, you
are free to purchase the products of other companies.  Right?  Or is
someone forcing you to purchase Nikon's software?

Really, what exactly is the problem here?  If anything, the very
existance of Dave Coffin's "dcraw" makes the ranting Reichmann, et al,
look very kookish, and rendering the entire "OpenRaw" issue moot:  RAW
file formats are as open as can be, _despite_ the best efforts of
Nikon.
Alan Browne - 26 May 2005 21:26 GMT
> It is not in Nikon's interest to tell everyone their innovations in
> (say) automatic white balance, or in Canon's interest to spill their
> beans re: (say) fancy device physics for optimal bias estimation in
> long exposure images.

No.  They give away all the data to "qualified" software developers.
They are not protecting anything to a degree where it would remain a
secret for more that a week or so.  That it can be cracked easilly by
writers of sw such as dcraw are testimony to the fact that it is pointless.

> If you don't like Nikon's or Canon's policies about any of this, you
> are free to purchase the products of other companies.  Right?  Or is
> someone forcing you to purchase Nikon's software?

Yes, Nikon.  For a person with 000's of dollars in Nikon glass, being
forced to purchase more beyond a camera is simple wallet gouging.

> Really, what exactly is the problem here?  If anything, the very
> existance of Dave Coffin's "dcraw" makes the ranting Reichmann, et al,
> look very kookish, and rendering the entire "OpenRaw" issue moot:  RAW
> file formats are as open as can be, _despite_ the best efforts of
> Nikon.

The point is to stop these idiotic and pointless 'races' in the future.
 When you buy a film camera, it works with the specified film.  No
special to Nikon processes are required.  Nikon are now requiring
special to type processes for their digital cameras.

Signature

-- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm
--        r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm
--      [SI] gallery & rulz: http://www.pbase.com/shootin
--                   e-meil: Remove FreeLunch.

DoN. Nichols - 27 May 2005 03:20 GMT
    [ ... ]

>> Really, what exactly is the problem here?  If anything, the very
>> existance of Dave Coffin's "dcraw" makes the ranting Reichmann, et al,
[quoted text clipped - 6 lines]
>special to Nikon processes are required.  Nikon are now requiring
>special to type processes for their digital cameras.

    When you buy a film camera, you are locked into the physical
format of the film.  If you want to shoot on some Royal-X Pan (very high
speed B&W film -- probably now long dead), it was available in 120 and
620 formats, but *not* in 35mm. (For good reason, as the grain would
have been excessive in 35mm.)

    Using 120 roll film or 4x5 sheet film in a 35mm camera is not
practical (though there were some strange adaptors in existence,
including one to take a 35mm image from the back of a Nikon F and expand
it to expose a 4x5 sheet film, or a 4x5 Polaroid (usually 3000 ASA).

    And to use other films, such as Infrared Ektachrome in your 35mm
SLR, you have to put some rather extreme filtration on it -- another
example of special processing.  (Granted, the filters are available form
other vendors, but the basic fact that your film camera will not work
with this film without special processing (the filters) remains.

    As for another phase of the same problem, when you buy a given
film, you may be required to buy a certain set of chemicals to process
that film -- much equivalent to requiring special software to process a
RAW format.  (Except that with the digital cameras, you are not *forced*
to use the RAW format -- the camera will happily provide you with JPEGs
(the equivalent of carrying the film to the corner drugstore for
processing.)

    But you could not process Kodak Ektachrome with Agfa chemicals
or Ansco chemicals.  For that matter, you could not process Kodak's
Kodachrome with Kodak's Ektachrome chemicals -- let alone processing
Kodacolor.

    Ektachrome even changed processes over time.  Sometimes the new
chemical set would work with the older film, and sometimes not.  I know
this, because I used to develop my own Ektachrome a *lot* -- from
perhaps around 1962 on through about 1976, when I got married, and was
working for a government organization which prohibited my bringing
cameras on base with me, so my photo work pretty much stopped until I
retired, and then the digital cameras came along.

    And there was at least one very high resolution B&W film which
required a special developer (was the maker H&W?  I used it -- once.)

    Granted -- for most B&W films, you could use a fairly wide range
of chemicals from different vendors to develop it --- including mixing
your own from old formulas -- or from knowing enough about the process
to devise your own.  (There was not enough information about the slide
films to allow this freedom.  And it is a lot harder to pick apart the
photochemistry of a transparency film than an image format.)

    But -- I don't see how the situation with RAW files is any worse
than these limitations from the film days.  And in some ways, it is a
*lot* better.  If you process a RAW file with the wrong software, the
worst that will happen is that you won't get anything useful.  The
original RAW file can be saved until you get the *right* software.

    Note that I prefer to do my image processing on unix computers,
which are *not* supported by either Kodak or Cannon (or any of the other
digital SLR makers, AFIK).  For this reason, I am quite pleased to have
Dave Coffin's "dcraw" software available.

    Yes -- having all the vendors use a common format would be
convenient (other than JPEG, which I detest as a lossy algorithm), but
I'll expect that to happen about the time that all film maker's color
processes are interchangeable. :-)

    Enjoy,
        DoN.
Signature

Email:   <dnichols@d-and-d.com>   | Voice (all times): (703) 938-4564
    (too) near Washington D.C. | http://www.d-and-d.com/dnichols/DoN.html
          --- Black Holes are where God is dividing by zero ---

John Francis - 27 May 2005 04:07 GMT
>    But -- I don't see how the situation with RAW files is any worse
>than these limitations from the film days.

In the film days, you could buy a different (relatively cheap) film
to use in your (relatively expensive) camera.  Nowadays the limitation
is embedded in the expensive component, and you can't avoid it.
Barry Pearson - 27 May 2005 08:46 GMT
[snip]
>     But -- I don't see how the situation with RAW files is any worse
> than these limitations from the film days.  And in some ways, it is a
> *lot* better.  If you process a RAW file with the wrong software, the
> worst that will happen is that you won't get anything useful.  The
> original RAW file can be saved until you get the *right* software.

I think the analogy with film can be taken too far. For example, DNG
could be more usefully compared with the developed negative, rather
than the undeveloped negative. Then we would be talking about
standardisation of film sizes to enable the negative to be processed
"anywhere" and by lots of enlargers and scanning and printing
equipment.

For asset management systems that handle many Raw formats, or decades
in the future when we want to process old Raw files, it may be hard to
"get the *right* software".

Sometimes we should just ask "what do we, as photographers or users of
photographs, want?" We surely wouldn't want lots of variants of small
negatives, nearly but not quite 35mm / 2 x 8 sprockets / 24mm x 36mm
image size. That is what we have with Raw formats at the moment.

(I'll pass over the fact that the DNG logo has 2 x 7 sprockets, not 2 x
8!)

>     Note that I prefer to do my image processing on unix computers,
> which are *not* supported by either Kodak or Cannon (or any of the other
> digital SLR makers, AFIK).  For this reason, I am quite pleased to have
> Dave Coffin's "dcraw" software available.
[snip]

You've just made the case for OpenRAW , and even DNG. Dave Coffin's
work is valuable, but it should never have been necessary. You should
be able to obtain a variety of Unix-based converters, etc, which were
based on proper specifications.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/
Alan Browne - 27 May 2005 14:38 GMT
> (I'll pass over the fact that the DNG logo has 2 x 7 sprockets, not 2 x
> 8!)

Cropped Digital Negative.

;-)

Signature

-- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm
--        r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm
--      [SI] gallery & rulz: http://www.pbase.com/shootin
--                   e-meil: Remove FreeLunch.

RichA - 27 May 2005 02:55 GMT
>> > activity -- or even the media itself who regularly troll their audience
>> > with prurient, irrelevant, or just plain false information.
[quoted text clipped - 22 lines]
>file formats are as open as can be, _despite_ the best efforts of
>Nikon.

Here we have an example of a prime case of the toadying
"company man" not concerned with customers.  It's a disease
that afflicts many people who work in companies over 1000 people,
or those who fantasize that their own work is more valuable than
good customer service.
-Rich
Jeremy Nixon - 27 May 2005 05:41 GMT
> Reichman, et al, say they don't want "trade secrets" revealed, but they
> are woefully clueless in that their demands for documentation of these
> files is exactly that.

That's nonsense.  No one is asking Nikon for the process by which white
balance is arrived at, only the conclusion.  If that could lead to
reverse engineering their process, then the fact that you can get it
using Nikon's software would lead to the exact same eventuality.

Signature

Jeremy  |  jeremy@exit109.com

Owamanga - 26 May 2005 19:29 GMT
>http://www.luminous-landscape.com/essays/raw-flaw.shtml

Chicken liken twaddle.

"Aggh! the sky is falling!"

http://www.ongoing-tales.com/SERIALS/oldtime/FAIRYTALES/chicklicken.html

--
Owamanga!
http://www.pbase.com/owamanga
Alan Browne - 26 May 2005 20:33 GMT
>>http://www.luminous-landscape.com/essays/raw-flaw.shtml
>
> Chicken liken twaddle.

Open RAW is best for everyone in the long term.  There's no excuse for
any data in the image created by the camera you own to be obfuscated or
encrypted.

Signature

-- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm
--        r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm
--      [SI] gallery & rulz: http://www.pbase.com/shootin
--                   e-meil: Remove FreeLunch.

eawckyegcy@yahoo.com - 26 May 2005 21:14 GMT
> Open RAW is best for everyone in the long term.

48V twisted pair POTS is a "standard" invented at the dawn of the
telephone era.  There are many engineers who wish we could rip the
entire mess out of the ground, off the poles and start over again.
Offer praises to Allah or whoever that cell phones, VOIP and the rest
of it are doing the job indirectly.

Be very careful what you wish for.
Alan Browne - 26 May 2005 21:36 GMT
>>Open RAW is best for everyone in the long term.
>
[quoted text clipped - 3 lines]
> Offer praises to Allah or whoever that cell phones, VOIP and the rest
> of it are doing the job indirectly.

Backward compatibilty is not compromised by forward progress where data
formats are concerned.  Formats can get ever richer without s/w losing
the ability to read old data correctly.  That is if s/w and formats are
designed and maintained correctly.

Or, if you prefer, open RAW does not mean 'cast in concrete'.  Each OEM
can do as he likes as long as the format is readable by all, and special
data sections are clearly documented.

Regarding the olde telephone analog standard it has served extremely
well for a very long time.  Real engineers are too practical to "wish"
for solutions that don't make economic sense.  They adapt instead to
growth without leaving anything of value behind.  This backward
compatibilty 'tax' is far cheaper than "ripping the entire mess out of
the ground..."

Signature

-- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm
--        r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm
--      [SI] gallery & rulz: http://www.pbase.com/shootin
--                   e-meil: Remove FreeLunch.

eawckyegcy@yahoo.com - 26 May 2005 22:06 GMT
>> 48V twisted pair POTS is a "standard" invented at the dawn of the
>> telephone era.  There are many engineers who wish we could rip the
[quoted text clipped - 4 lines]
> Backward compatibilty is not compromised by forward progress where data
> formats are concerned.

Then you must agree that OpenRAW is without a legitimate function.
What's good for the goose, etc.

> Or, if you prefer, open RAW does not mean 'cast in concrete'.  Each OEM
> can do as he likes as long as the format is readable by all, and special
> data sections are clearly documented.

As I said, this will in many instances necessarily reveal trade secrets
or other facets of the technology that the manufacturers would likely
be unwilling to disclose.  And if part of the OpenRAW is the signing of
an NDA, then can we honestly call it "open"?

> Regarding the olde telephone analog standard it has served extremely
> well for a very long time.  Real engineers are too practical to "wish"
> for solutions that don't make economic sense.

If some djinn were to remove the need to work against the crazy POTS
nonsense, many communications engineers would be profoundly thankful.
Indeed, they must be:  look at where all the innovation is occuring
today re: telephony.  I can cite similar software and hardware
examples.  _NO ONE_ likes to deal with legacy systems.

> This backward compatibilty 'tax' is far cheaper than "ripping the
> entire mess out of the ground..."

You are forgetting or ignoring the cost of innovations that are simply
unimplementable within the legacy framework.  It is for this and other
reasons that the sensible person does not want camera manufacturers
constrained by some Adobe or OpenRAW or otherwise committee drafted
multi-volume 2356 page standard written in dense legalese.
Alan Browne - 26 May 2005 22:37 GMT
>>>48V twisted pair POTS is a "standard" invented at the dawn of the
>>>telephone era.  There are many engineers who wish we could rip the
[quoted text clipped - 7 lines]
> Then you must agree that OpenRAW is without a legitimate function.
> What's good for the goose, etc.

I don't follow that one.  What makes sense is that what gets recorded
onto the memory card is the property of the photographer or employer,
not the camera OEM.

>>Or, if you prefer, open RAW does not mean 'cast in concrete'.  Each OEM
>>can do as he likes as long as the format is readable by all, and special
[quoted text clipped - 4 lines]
> be unwilling to disclose.  And if part of the OpenRAW is the signing of
> an NDA, then can we honestly call it "open"?

Since these trade secrets have a half life in the wild of about 3 days,
they are no more useful to the OEM's than encryption has been for the
DVD producers.  NDA's don't protect anything of this nature.

"Trade Secret"?  Why hasn't the dcraw author been sued into
homelessness?  If he can do it, the folks at competing OEM's can do it
just as quick.

>>Regarding the olde telephone analog standard it has served extremely
>>well for a very long time.  Real engineers are too practical to "wish"
>>for solutions that don't make economic sense.
>
> If some djinn were to remove the need to work against the crazy POTS
> nonsense, many communications engineers would be profoundly thankful.

But that's not engineering.  The same could be said about many
infrastructures from roadways to airways.  If we were to design "World
2.0" do you think we would have LA, Mexico and Cairo (to name very few).

> Indeed, they must be:  look at where all the innovation is occuring
> today re: telephony.  I can cite similar software and hardware
> examples.  _NO ONE_ likes to deal with legacy systems.

If it is economically sensible to do so, it is done.  That is
engineering in large part.  "Like" has little to with it.

>>This backward compatibilty 'tax' is far cheaper than "ripping the
>>entire mess out of the ground..."
[quoted text clipped - 4 lines]
> constrained by some Adobe or OpenRAW or otherwise committee drafted
> multi-volume 2356 page standard written in dense legalese.

See above, re: World 2.0.

If that's so, why are they all committe members, participants and
signatories to various other standards?

Feel free to have the last word ... this is it for me.

Cheers,
Alan

Signature

-- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm
--        r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm
--      [SI] gallery & rulz: http://www.pbase.com/shootin
--                   e-meil: Remove FreeLunch.

Alan Browne - 26 May 2005 22:47 GMT
> infrastructures from roadways to airways.  If we were to design "World
> 2.0" do you think we would have LA, Mexico and Cairo (to name very few).

oops.  Meant "Mexico City".

Signature

-- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm
--        r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm
--      [SI] gallery & rulz: http://www.pbase.com/shootin
--                   e-meil: Remove FreeLunch.

eawckyegcy@yahoo.com - 26 May 2005 23:20 GMT
>>>Backward compatibilty is not compromised by forward progress where data
>>>formats are concerned.
[quoted text clipped - 3 lines]
>
> I don't follow that one.

Hey, you made the claim.  If your statement is true, then the camera
makers are free to invent whatever new file formats they like ("forward
progress") since "backward compatibility is not compromised".  And with
such freedom, what is OpenRAW's purpose?

>                           What makes sense is that what gets recorded
> onto the memory card is the property of the photographer or employer,
> not the camera OEM.

Sounds like you need an IP attorney, not an OpenRAW standard.

>> As I said, this will in many instances necessarily reveal trade secrets
>> or other facets of the technology that the manufacturers would likely
[quoted text clipped - 4 lines]
> they are no more useful to the OEM's than encryption has been for the
> DVD producers.  NDA's don't protect anything of this nature.

Yes, yes, the manufacturers can make idiots of themselves.  Why are you
so concerned about this?

> "Trade Secret"?  Why hasn't the dcraw author been sued into
> homelessness?

He hasn't published anything particularly sensitive yet?  But even if
he did, trade secrets, in the absence of a contract, have basically no
legal protections at all.

>                If he can do it, the folks at competing OEM's can do it
> just as quick.

Reverse engineering is an honoured tradition.  What better compliment
can you offer the manufacturer?

>> If some djinn were to remove the need to work against the crazy POTS
>> nonsense, many communications engineers would be profoundly thankful.
>
> But that's not engineering.

Of course it is.  There are many situations, particularly in software,
where one invents brand new stuff all the time.  No legacy constraints.
It is a most refreshing experience.

>                                 The same could be said about many
> infrastructures from roadways to airways.

Your point?

>                                             If we were to design "World
> 2.0" do you think we would have LA, Mexico and Cairo (to name very few).

I honestly couldn't say.  And really, is this relevant?

>> Indeed, they must be:  look at where all the innovation is occuring
>> today re: telephony.  I can cite similar software and hardware
>> examples.  _NO ONE_ likes to deal with legacy systems.
>
> If it is economically sensible to do so, it is done.  That is
> engineering in large part.  "Like" has little to with it.

"Like" has everything to do with it, since if nobody likes it, it won't
get done.  More importantly, though, if physical reality says "no", it
doesn't matter what you think.

>>>This backward compatibilty 'tax' is far cheaper than "ripping the
>>>entire mess out of the ground..."
[quoted text clipped - 6 lines]
>
> See above, re: World 2.0.

Ok, then, send Canon a check to cover their costs of abiding by the
Official OpenRAW Standard, or to make up for lost revenue to products
or services which they would like to produce, but which can not fit
into this standard.

> If that's so, why are they all committe members, participants and
> signatories to various other standards?

Oh, I could rant about this at some length.  But why should I entertain
your distractions?

> Feel free to have the last word ... this is it for me.

Thanks!
Barry Pearson - 27 May 2005 00:20 GMT
[snip]
> Ok, then, send Canon a check to cover their costs of abiding by the
> Official OpenRAW Standard, or to make up for lost revenue to products
> or services which they would like to produce, but which can not fit
> into this standard.
[snip]

The cost of abiding by a common Raw format would probably be small.
There would probably be a start-up cost - "climbing the learning
curve". (If I were a manager in a camera-making company, I would prefer
to manage the second camera in the company to use it, not the first!)

After that, it would probably be cheaper. There might be cost-savings
such as common bits of firmware that could be bought, re-used, etc. And
savings in learning about the format. There has been a lot of "wheel
re-inventing" up to now, with must surely add to cost, and perhaps
reduce quality.

No camera launched in the last year and a half would have needed a
change to a well-engineered common Raw format. The vast majority of
changes, such as more pixels, greater bit depth, etc, don't need
changes. So there may not be any such products and services that they
couldn't produce.

The one restriction that camera manufacturers would face is that they
could not then use encryption to coerce their customers to buy their
software.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/
Barry Pearson - 27 May 2005 00:51 GMT
[snip]
> As I said, this will in many instances necessarily reveal trade secrets
> or other facets of the technology that the manufacturers would likely
> be unwilling to disclose.  And if part of the OpenRAW is the signing of
> an NDA, then can we honestly call it "open"?

Publicising current formats may reveal some things that manufacturers
would not like revealed. Although that may be just the design quality
of their Raw file formats! But interface specifications don't normally
reveal much about the inside of the box. I would expect the formats to
identify the sensor configuration, but the manufacturers are typically
willing, sometimes eager, to tell the world about that anyway.

Use of a common Raw format would reveal even less. At the moment, the
DNG format caters for Bayer sensors, Fujifilm SR sensors, Sony 4-colour
sensors, and Foveon sensors. (Possibly lots more configurations could
also be handled). If you can handle all of those with a single
specification, then using that specification isn't revealing much about
your technology. Formats designed specifically for your technology are
more likely to be revealing, if only accidentally.

[snip]
> You are forgetting or ignoring the cost of innovations that are simply
> unimplementable within the legacy framework.  It is for this and other
> reasons that the sensible person does not want camera manufacturers
> constrained by some Adobe or OpenRAW or otherwise committee drafted
> multi-volume 2356 page standard written in dense legalese.

What innovations are those? No new cameras in the last year and a half
would have needed a change to a common Raw format. The vast majority of
innovations are not visible in details of the Raw file format.

The DNG specification is 50 pages long, and a lot of that is white
space. It builds on TIFF/EP (the draft was 65 pages). But the camera
manufacturers used that anyway! Typical proprietary Raw formats are
already based on TIFF/EP. (And TIFF/EP refers to others). At the
moment, internally the manufacturers must use these external
specifications plus their own internal usage specifications. DNG, or
other common Raw format, would replace some internal specification.

Any photographer concerned with the future health of top-end digital
photography ought to want camera manufacturers to use a common Raw
format. And DNG is suitable for that purpose.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/
eawckyegcy@yahoo.com - 27 May 2005 01:24 GMT
> Publicising current formats may reveal some things that manufacturers
> would not like revealed. Although that may be just the design quality
> of their Raw file formats!

I can't comment on Nikon stuff, since I'm a Canon sort of person.

Canon 10D:  a bizarre CIFF, for which they published the specifications
for the "superstructure" of this file.  It's a kind of weird TIFF;  you
can, in fact, TIFFize a CIFF.  There are two images in this file:  the
JPEG and an strangely encoded raw sensor dump.  That Dave Coffin
managed to figure it out speaks for his abilities as a Reverse
Engineer.

Canon 1DMkII: it's a straight TIFF, with EXIF markup.  Three images
embedded:  a flat RGB thumbnail, a 1536 pixel wide JPEG (used for
in-camera editing), and the lossless JPEG encoded sensor dump.

The main mysteries for both are not the bulk format, but the
undocumented tags, specifically the "EXIF MakerNote" as well as the
various non-standard TIFF tags.

>                              But interface specifications don't normally
> reveal much about the inside of the box. I would expect the formats to
> identify the sensor configuration, but the manufacturers are typically
> willing, sometimes eager, to tell the world about that anyway.

I have reverse engineered a fair amount of the 10D's undocumented tags
(and some of the 1DMkII's -- too busy).  There is one that does indeed
lay out of the geometry of the sensor (how large, where the "image"
starts, and so forth).  There are other tags that detail the active AF
sensor bitmap, various light meter readings, exposure computations and
so forth.  Just a few days ago, I found (by accident) the default
luminance curve in the 1DMkII CR2 file.

Probably the most interesting aspect is one that is relevant here:  the
sensor has a 64 column bitmap "black" pixels on the left side of the
image.  The usual assumption is that these are used for bias level
stuff.  However, there is unusual stuff occuring here when the
exposures grow to 30 seconds or more ... the 64 columns splits into
distinct 16 column set and a 48 column set, and these columns are
(apparently) used to produce a much better bias estimate than would be
obtained from simple averaging.

This is precisely the sort of detail that Canon would almost certainly
_not_ want to be generally known (it took me a while to figure it out),
not just that it exists, but that someone might be able to figure out
the device physics and go from there.  If this is such a thing, though,
there are almost certainly others awaiting.

Or maybe everyone knows this stuff because it is a Canon patented
process.

> Use of a common Raw format would reveal even less.

Suppose the above 16/48 column stuff was a Canon "trade secret" they
would prefer to keep.  How could a standard format encompass such
sensor behaviour without revealing the secret?  Or suppose that
everyone knows this -- it is a Canon patent.  Unless Canon licenses
this patent for zero cost (which we can assume will be a
zero-probability event), no one can use it even if it was fully
documented to the last bit.

> Formats designed specifically for your technology are
> more likely to be revealing, if only accidentally.

I suggest this is more common than people think.  The "MakerNote"
information is tightly coupled to the hardware, at least it is for
Canon equipment.
Jeremy Nixon - 27 May 2005 05:36 GMT
> Suppose the above 16/48 column stuff was a Canon "trade secret" they
> would prefer to keep.  How could a standard format encompass such
> sensor behaviour without revealing the secret?

Does not the fact that DNG can *already* handle it indicate that this
"problem" is nonexistent?  Does not the fact that DNG can fully handle
data from cameras that didn't exist at the time its code was written
mean that this has no actual basis in reality?

Signature

Jeremy  |  jeremy@exit109.com

eawckyegcy@yahoo.com - 27 May 2005 17:06 GMT
>> Suppose the above 16/48 column stuff was a Canon "trade secret" they
>> would prefer to keep.  How could a standard format encompass such
>> sensor behaviour without revealing the secret?
>
> Does not the fact that DNG can *already* handle it indicate that this
> "problem" is nonexistent?

Cite the relevant sections in the DNG specification which detail
exactly this -- that is, how to handle the 16/48 column stuff -- and
you have made a point.
Jeremy Nixon - 27 May 2005 18:20 GMT
> Cite the relevant sections in the DNG specification which detail
> exactly this -- that is, how to handle the 16/48 column stuff -- and
> you have made a point.

It doesn't have to specify such a thing.  That's not what DNG is, or what it
does.  It doesn't have to handle, or know about, every technical difference
in every sensor design.  Indeed, it proves that using a different RAW format
for every camera model is completely unnecessary.

Signature

Jeremy  |  jeremy@exit109.com

eawckyegcy@yahoo.com - 27 May 2005 19:16 GMT
>> Cite the relevant sections in the DNG specification which detail
>> exactly this -- that is, how to handle the 16/48 column stuff -- and
>> you have made a point.
>
> It doesn't have to specify such a thing.

Yes it does.  If the native camera format can do things that the DNG
can't, why use the DNG?

> That's not what DNG is, or what it does.

Shifting the goal-posts now?  OpenRAW wants full, complete,
documentation.

>        It doesn't have to handle, or know about, every technical difference
> in every sensor design.

If it doesn't, then you lose capabilities that may be associated with
the sensor.  If the DNG removed the red channels from all RAW images,
would you use it?

>                           Indeed, it proves that using a different RAW format
> for every camera model is completely unnecessary.

I have data that suggests that Canon cameras have unique black level
estimation for long exposure images.  The DNG format does not allow for
such behaviour (or at least you have not presented me with a citation
-- and I couldn't find one when the DNG spec was initially released).

Now if the DNG can't handle that, what do you recommend I do with my
multi-minute minute exposure CR2 files I have?
Jeremy Nixon - 27 May 2005 19:32 GMT
>> It doesn't have to specify such a thing.
>
> Yes it does.  If the native camera format can do things that the DNG
> can't, why use the DNG?

It's not something the format, native or DNG, "does".

>> It doesn't have to handle, or know about, every technical difference
>> in every sensor design.
>
> If it doesn't, then you lose capabilities that may be associated with
> the sensor.

No, you don't.

> I have data that suggests that Canon cameras have unique black level
> estimation for long exposure images.  The DNG format does not allow for
[quoted text clipped - 3 lines]
> Now if the DNG can't handle that, what do you recommend I do with my
> multi-minute minute exposure CR2 files I have?

It is not something DNG needs to "handle".

Try taking your files, converting them to DNG, and loading them into
Camera Raw.  It'll work.

Signature

Jeremy  |  jeremy@exit109.com

eawckyegcy@yahoo.com - 27 May 2005 19:44 GMT
>> I have data that suggests that Canon cameras have unique black level
>> estimation for long exposure images.  The DNG format does not allow for
[quoted text clipped - 5 lines]
>
> It is not something DNG needs to "handle".

Then the DNG is useless re: the above situation, since the native
CR2/CRW files (and their decoders from Canon) handle it just fine.

> Try taking your files, converting them to DNG, and loading them into
> Camera Raw.  It'll work.

Something will happen, but probably not the _optimal_ thing.  Maybe you
have lesser goals, but I paid alot for my camera, and want my images
given as good a treatment as possible.  If DNG can't do it, why should
I use it?
Jeremy Nixon - 27 May 2005 19:55 GMT
>> It is not something DNG needs to "handle".
>
> Then the DNG is useless re: the above situation, since the native
> CR2/CRW files (and their decoders from Canon) handle it just fine.

You don't understand what DNG is.  The native RAW files don't have to
"handle" it, either.  One has nothing to do with the other.

Signature

Jeremy  |  jeremy@exit109.com

eawckyegcy@yahoo.com - 27 May 2005 20:11 GMT
Jeremy Nixon babbles:

> The native RAW files don't have to "handle" it, either.

Once again, nitwit:  I have data that hints that Canon has a special
black-level estimator for long exposure images.

This estimator, if it exists, would be built into Canon CR2 file
decoders.

None of this is known to DNG. (I am still awaiting a DNG spec
citation.)

Thus no current DNG decoder uses this special estimator.

Given this context, why should I use DNG over CR2?

Oh, you might argue, but Canon could make a DNG decoder that knows all
about that.  Shall I respond to this now, or can you figure out the
problem on your own?
Jeremy Nixon - 27 May 2005 20:41 GMT
>> The native RAW files don't have to "handle" it, either.
>
> Once again, nitwit:  I have data that hints that Canon has a special
> black-level estimator for long exposure images.

Good for you.

> This estimator, if it exists, would be built into Canon CR2 file
> decoders.
>
> None of this is known to DNG. (I am still awaiting a DNG spec
> citation.)

Then I'm awaiting an explanation of why DNG should have to know about it.
Given that it *already works* in DNG *right now*, I think it's pretty
clear that it doesn't, any more than it has to know about every other
camera that comes out.

I'm currently processing my camera's RAW files after converting to DNG.
My camera didn't exist at the time the version of Camera Raw I'm using
was written.  Yet, somehow, it works perfectly.  How do you suppose that
is?  How could DNG possibly "know about" my camera's unique RAW format?

Because it doesn't have to, of course.

> Given this context, why should I use DNG over CR2?

You shouldn't.  No one is trying to make you.  But neither should you
argue against DNG based on faulty understanding of what it is.  You
are assuming that something Canon is doing would require special
consideration in the RAW file format; it does not require any such
thing.

If something Canon is doing results in their RAW conversion software
doing a better job than third-party software does, that is a completely
different thing and it has nothing, at all, to do with the RAW file format.

Signature

Jeremy  |  jeremy@exit109.com

Barry Pearson - 27 May 2005 23:41 GMT
[snip]
> I'm currently processing my camera's RAW files after converting to DNG.
> My camera didn't exist at the time the version of Camera Raw I'm using
> was written.  Yet, somehow, it works perfectly.  How do you suppose that
> is?  How could DNG possibly "know about" my camera's unique RAW format?
>
> Because it doesn't have to, of course.
[snip]

Thomas Knoll told me that no camera of the last year and a half would
have needed a change to DNG, had DNG existed that long ago.

Typically, to support a new camera, it isn't the DNG specification that
needs to change. Instead, what has to happen is that something has to
write, into specified places in the DNG file, specific information
about the camera model. (In addition to the sensor data for the
photograph, of course!) Then the Raw processing software, ACR or
something else, can extract this specific information instead of having
built-in knowledge of the camera.

I believe it works like this. Suppose I launch a new camera tomorrow,
that writes Raw files with a BCP extension. I also release some
software to process those Raw files. The camera has some existing
sensor configuration, such as Bayer, or Fujifilm SR, or Sony 4-color,
or Foveon, and writes just the sensor data to the BCP file. My software
obviously knows what the sensor configuration is, and interprets the
sensor data accordingly. This also includes interpreting the sensor
values according to the colour temperature.

No other software has a clue how to interpret the sensor data. It won't
even know what the sensor configuration is, and that is obviously
vital, even before you worry about the colour temperature and other
things. All Raw processing software will there have to give up on this
BCP file format. Obviously, ACR 2.4 in Photoshop CS has to give up,
because its built-in tables don't cover this camera.

Now Adobe releases a new version of the DNG Converter that knows about
this camera. This DNG Converter writes the sensor data to the DNG file,
of course. And also writes fields that describe the sensor
configuration, and how to interpret the colour temperature, etc. So now
the DNG file is "self contained". Some fields provide the sensor data
for the photograph. Others identify how to interpret that data.
(Metadata).

Now, when ACR 2.4, or any other Raw processor that handles DNG, reads
that file, it doesn't need its own data to tell it how to interpret the
data for this camera. It just extracts that from the DNG file. It can
handle cameras it has never before come across, because the DNG file
itself tells it how to. (I think the DNG Converter writes colour
calibration fields for 2 colour tempertaures, one tungsten and the
other daylight, to the file. This enables ACR to do its white balance
handling).

I'm still hazy about some of the details. But what I've said above is
consistent with everything I've learned and tested so far. I have
converted D2X and 350D Raw files into DNG, and processed them in ACR
2.4 under CS. CS Browser shows thumbnails for the DNG files just as it
would any Raw files that it recognised, and ACR 2.4 can process them.
But it just shows place-holders for the original Raw files, and refuses
to open the latter because it doesn't recognise the format.

It is this concept of being "self contained", with the sensor data for
the image, plus metadata identifying how to interpret the sensor data,
that makes DNG very powerful. Someone could release a photo-editor
tomorrow that only accepted DNG, and it would be able to process all 77
cameras that the DNG Converter could handle. Someone could release a
new camera tomorrow that recorded DNG, and immediately all Raw
processors that handled DNG properly would support that camera. And in
decades to come, DNG processors will be able to handle old DNG files
that have images taken using cameras that no one knows about anymore.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/
Jeremy Nixon - 28 May 2005 01:32 GMT
> The camera has some existing sensor configuration, such as Bayer, or
> Fujifilm SR, or Sony 4-color,

Indeed, I don't think the Sony 4-color requires any special consideration
in DNG, as per some statements I've seen from Thomas Knoll.  It's still a
Bayer pattern, and the particular colors in use apparently don't matter
since they are specified in the DNG file.

> I'm still hazy about some of the details. But what I've said above is
> consistent with everything I've learned and tested so far. I have
[quoted text clipped - 3 lines]
> But it just shows place-holders for the original Raw files, and refuses
> to open the latter because it doesn't recognise the format.

I've used it for my D2x files, too, in ACR 2.4, with complete success.
It still does a better job than either dcraw or Nikon's own software,
despite having been written before the D2x existed.

Signature

Jeremy  |  jeremy@exit109.com

eawckyegcy@yahoo.com - 28 May 2005 00:27 GMT
>> Once again, nitwit:  I have data that hints that Canon has a special
>> black-level estimator for long exposure images.
>
> Good for you.

I know, it is.  What data do you have, homeboy?

> > This estimator, if it exists, would be built into Canon CR2 file
> > decoders.
[quoted text clipped - 3 lines]
>
> Then I'm awaiting an explanation of why DNG should have to know about it.

Ohhhhh, I dunno, in order to make a better black level estimate?

> Given that it *already works* in DNG *right now*, I think it's pretty
> clear that it doesn't, any more than it has to know about every other
> camera that comes out.

Well, maybe your code is making non-optimal estimates.  Oh my.  Oh me.

> I'm currently processing my camera's RAW files after converting to DNG.
> My camera didn't exist at the time the version of Camera Raw I'm using
> was written.  Yet, somehow, it works perfectly.  How do you suppose that
> is?  How could DNG possibly "know about" my camera's unique RAW format?
>
> Because it doesn't have to, of course.

Your converter can make as many assumptions as it likes, of course.
Maybe you like guessing:  I like knowing.

> > Given this context, why should I use DNG over CR2?
>
> You shouldn't.  No one is trying to make you.  But neither should you
> argue against DNG based on faulty understanding of what it is.

Take your straw-men and shove 'em.  Now answer the question as posed:
if Canon software can render a better image than a DNG decoder, why use
the DNG decoder?

>                                                                 You
> are assuming that something Canon is doing would require special
> consideration in the RAW file format; it does not require any such
> thing.

I am not assuming Canon is doing something:  _I KNOW THEY ARE_.
Remember the "data" that is "good for me"?

> If something Canon is doing results in their RAW conversion software
> doing a better job than third-party software does, that is a completely
> different thing and it has nothing, at all, to do with the RAW file format.

Canon's converters will do a better job becaues they have knowledge of
(a) their hardware, but more importantly, (b) specific information in
their CR2 files which directs superior handling of the raw image data.
SHOCK OF SHOCKS:  some of this information may not be encodable in the
DNG file!
Jeremy Nixon - 28 May 2005 01:27 GMT
>> Then I'm awaiting an explanation of why DNG should have to know about it.
>
> Ohhhhh, I dunno, in order to make a better black level estimate?

DNG does not make a black level estimate at all, so making a better one
is really not relevant.

The information you're using to make that black level estimate can be
preserved in a DNG file, though, so that software that knows how to use
it will be able to.

> Take your straw-men and shove 'em.  Now answer the question as posed:
> if Canon software can render a better image than a DNG decoder, why use
> the DNG decoder?

Absolutely no reason at all.  None whatsoever.  No one is trying to say
that you should do that.  Indeed, if you prefer the results you get with
Canon's software, I would say that you should use it.

> I am not assuming Canon is doing something:  _I KNOW THEY ARE_.
> Remember the "data" that is "good for me"?

You know Canon is doing something; you assume that it will require
specific support in the RAW file format, and that assumption is not
correct.

> Canon's converters will do a better job becaues they have knowledge of
> (a) their hardware, but more importantly, (b) specific information in
> their CR2 files which directs superior handling of the raw image data.

If Canon's converters do a better job, then go ahead and use them.

> SHOCK OF SHOCKS:  some of this information may not be encodable in the
> DNG file!

Except that it is, so that doesn't apply.  It *is* possible to do something
that can't be encoded by the existing DNG specification, but Canon hasn't
done so, nor has any camera since (I believe) the Fuji 2-level sensor
thing.

Signature

Jeremy  |  jeremy@exit109.com

JPS@no.komm - 28 May 2005 02:02 GMT
>Then I'm awaiting an explanation of why DNG should have to know about it.
>Given that it *already works* in DNG *right now*,

What do you mean by "already works"?  How do you know the conversion
couldn't be better?  DNG does not keep all black pixels; it has a tag
that *may* report the difference for each horizontal and vertical black
line from the global blackpoint it writes to the file.
Signature


<>>< ><<> ><<> <>>< ><<> <>>< <>>< ><<>
  John P Sheehy         <JPS@no.komm>

><<> <>>< <>>< ><<> <>>< ><<> ><<> <>><
JPS@no.komm - 28 May 2005 01:59 GMT
>Jeremy Nixon babbles:
>
[quoted text clipped - 10 lines]
>
>Thus no current DNG decoder uses this special estimator.

It is possible that the DNG ENcoder takes it into account, though.

That's the way DNG 3.1 deals with 20D banding, I think; when it writes
the DNG, it alters the RAW data so that it needs the same blackpoint for
every line.

>Given this context, why should I use DNG over CR2?
>
>Oh, you might argue, but Canon could make a DNG decoder that knows all
>about that.  Shall I respond to this now, or can you figure out the
>problem on your own?

Or, Canon might allow for the camera to record this information, but
Canon never actually uses it; just like they don't use the individual
lines' black data in their RAW processors, but DNG 3.1 *does*.
Signature


<>>< ><<> ><<> <>>< ><<> <>>< <>>< ><<>
  John P Sheehy         <JPS@no.komm>

><<> <>>< <>>< ><<> <>>< ><<> ><<> <>><
Owamanga - 27 May 2005 20:15 GMT
>>> It is not something DNG needs to "handle".
>>
[quoted text clipped - 3 lines]
>You don't understand what DNG is.  The native RAW files don't have to
>"handle" it, either.  One has nothing to do with the other.

Then explain, because your comments have me confused now.

Either DNG is:

1) Simply a container, imbecilically storing the RAW file within
without any comprehension as to its content. Basically no better than
zipping the damn RAW file up.

2) A new representation of the data within the RAW file, which
therefore must either understand and re-represent every aspect of the
original RAW file in an open format manner or be considered a lossy
storage method.

3) Some hybrid of the above, meaning it must be significantly larger
than the RAW file originally was.

Until DNG is so good, that in every case, once a RAW is converted to
DNG we can safely throw the original RAW file away and have lost
nothing, it'll have extremely limited appeal.

Even if Adobe's solution promises to be able to recreate the original
RAW file from the DNG, that's really zero steps better than a zip
file.

--
Owamanga!
http://www.pbase.com/owamanga
Barry Pearson - 27 May 2005 21:54 GMT
[snip]
> Either DNG is:
>
> 1) Simply a container, imbecilically storing the RAW file within
> without any comprehension as to its content. Basically no better than
> zipping the damn RAW file up.

No. Code can be written that uses the DNG specification to turn a
DNG-conformant file into a recognisable image, without needing to know
anything about the camera concerned. So DNG obviously has the necessary
semantics.

(DNG is a variant of TIFF/EP. So is the typical Raw file format.
Typically, much of the important data has the same meaning in both
formats).

> 2) A new representation of the data within the RAW file, which
> therefore must either understand and re-represent every aspect of the
> original RAW file in an open format manner or be considered a lossy
> storage method.

No. The essential image data is understand, but often not
re-represented. Some contents of Raw file formats are already likely to
conform to DNG, because they conform the recognised parts of TIFF/EP.
If a Raw format already has the sensor data in the required format, why
re-represent it? (The fact that a file format is proprietary and not
open doesn't mean that "every aspect of the original RAW file" is
closed, not open. Surely the bits that conform to TIFF/EP can be
considered to be open?).

In addition, DNG offers the option of including non-essential private
data that is not dictated by the specification. So it doesn't
understand "every aspect".

> 3) Some hybrid of the above, meaning it must be significantly larger
> than the RAW file originally was.

No. In all cases that I've tested, without embeding the original file,
the DNG version was smaller than the original version. But only when
losslessly compressed. So this is probably just a visible indication of
the relative effectiveness of the lossless compression used by DNG
compared with the compression used in the original format.

(Note that even a D2X NEF file saved by Nikon Capture 4.2.1, while
smaller than that output by the D2X camera, is larger than the DNG
version. I think DNG simply uses a very effective lossless compression
scheme).

> Until DNG is so good, that in every case, once a RAW is converted to
> DNG we can safely throw the original RAW file away and have lost
> nothing, it'll have extremely limited appeal.

No. I don't save the original after converting to DNG. I've been losing
things converting to DNG, such as the lens-model used for the shot. But
those things haven't been important enough to me to do otherwise. Lots
of other people appear to think likewise.

My guess is that DNG will be the format of choice for most Raw shooters
who use CS2 and ACR 3.1. It is so convenient. And with about 25
non-Adobe products supporting DNG, its use won't be confined to
Photoshop / ACR.

> Even if Adobe's solution promises to be able to recreate the original
> RAW file from the DNG, that's really zero steps better than a zip
> file.

No. One method of being able to recreate the original Raw file from the
DNG is to embed the original file in the DNG. Apart from the size
problem, this can be attractive. The file behaves, of course, just like
a DNG file, and can therefore be used in all cases where DNG would be
used. But the original can be extracted with bit-level accuracy at any
time.

Frankly, I haven't a clue whether the "unique black level estimation
for long exposure images" is catered for by the current version of DNG.
Since I don't use Canon, I have no need to think about it. Why couldn't
it be held in DNGPrivateData, or in MakerNote with MakerNoteSafety set
to 1 (safe)? If it can't, then Canon could ask for a new version. That
is why DNG has a good version scheme.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/
eawckyegcy@yahoo.com - 27 May 2005 23:01 GMT
> Frankly, I haven't a clue whether the "unique black level estimation
> for long exposure images" is catered for by the current version of DNG.

It isn't.

> Since I don't use Canon, I have no need to think about it.

www.google.com:  define:abstract

Nikon (or whoever) has likely has something that is being dropped on
the floor when you convert to DNG.  This is the basic point (that is
apparently lost on too many people):  without specific support for all
features present in a camera, DNG is necessarily lossy.

>                                                              Why couldn't
> it be held in DNGPrivateData, or in MakerNote with MakerNoteSafety set
> to 1 (safe)?

Because no standard DNG reader will know what to do with it, why
bother?

> If it can't, then Canon could ask for a new version.

Why should they?  Instead they can make a special reader which does
know what to do with the "PrivateData" ... but hey, wait a minute ...

> That is why DNG has a good version scheme.

The existence of "PrivateData" is the problem, not the lack or presence
of version numbering.

In short:

a) if no DNG standard reader can understand the "private" stuff, why
bother storing it at all?  In this case, DNG offers you nothing more
than what a TIFF file would.

b) if some DNG reader _can_ understand it, it must be a custom reader,
and thus one has just converted the "file format problem" into an
"application format problem".  If you have to keep track of reader X to
best deal with DNG's that come from camera X in order to get the most
out of the images, in what way would this be different from the current
situation?  In this case, DNG has given you nothing but an extra, lazy,
man in the middle that drops bits on the floor.

Note that in both cases the DNG (or OpenRAW or whatever) gives you
nothing.  Why, then, use it?
Jeremy Nixon - 27 May 2005 23:18 GMT
> Nikon (or whoever) has likely has something that is being dropped on
> the floor when you convert to DNG.  This is the basic point (that is
> apparently lost on too many people):  without specific support for all
> features present in a camera, DNG is necessarily lossy.

Except that this is not correct.  DNG does *not* need specific support
for all camera features in order to preserve them; indeed, that's part
of the whole point of the thing.

Signature

Jeremy  |  jeremy@exit109.com

eawckyegcy@yahoo.com - 28 May 2005 00:10 GMT
>> Nikon (or whoever) has likely has something that is being dropped on
>> the floor when you convert to DNG.  This is the basic point (that is
[quoted text clipped - 4 lines]
> for all camera features in order to preserve them; indeed, that's part
> of the whole point of the thing.

Even taken alone, your statement is nonsensical.  It is isomorphic to
"You don't need an arm in order to use an arm."

Start making sense, dude.
Barry Pearson - 28 May 2005 08:58 GMT
[snip]
> > Except that this is not correct.  DNG does *not* need specific support
> > for all camera features in order to preserve them; indeed, that's part
> > of the whole point of the thing.
>
> Even taken alone, your statement is nonsensical.  It is isomorphic to
> "You don't need an arm in order to use an arm."
[snip]

Let's step back to see what is being talked of here:

DNG is a specification of a file format. It can cater for a whole range
of sensor configurations, including Bayer-like, Foveon-like,
Fujifilm-like, Sony-like, and others. It has various parameters that
provide extra details for each of these.

Any particular DNG image file uses just the subset which this camera
needs, out of the vast range of possibilities. So that file is doing 2
important things, as far as this discussion is concerned. It identifies
those details of the camera that will be needed when processing the
image. And it supplies the image data itself.

It isn't important whether the DNG specification took this particular
camera model into account. What matters is that the specification has
sufficient richness to cater for this camera model. So a new camera
model that differs only in bits-per-pixel, or number of X and Y pixels,
or different look-up tables to transform the sensor data according to
colour temperature, can be catered for.

It appears to be the case that DNG specification version 1.1.0.0,
released in January, had sufficient richness for the 350D and D2X
details. In fact, probably version 1.0.0.0, released last September,
already had sufficient richness.

It is beyond doubt that ACR 2.4, released in January, already had the
code within it to read DNG files generated from D2X NEFs and 350D CR2s,
and process those images. I've tried this route, and I guess that by
now lots of other people have. I wonder whether ACR 2.3, released in
September, also had sufficient code within it to handle those DNG
files? I suspect it did, because there are only small differences
between versions 1.0.0.0 and 1.1.0.0.

Yet neither ACR 2.3 nor ACR 2.4 can read a D2X NEF file or a 350D CR2
file. It takes specific knowledge of a camera model to process Raw
files, and ACR 2.x didn't have that knowledge. (Neither, in fact, did
ACR 3.0).

This shows the power of DNG. You can write code to process files
conforming to particular versions of DNG, and you can cater
automatically for cameras that you haven't heard of, and that may not
have existed when you wrote the code. The advantage for the camera
manufacturer would be that their new camera is immediately supported by
all Raw processors that catered properly for the chosen version of DNG.
If a future D3X or 375D output in DNG version 1.1.0.0, then ACR 2.4
could process the images.

When new technology cannot be catered for by the current DNG version, a
new version, for example 2.0.0.0, will be needed. That presumably can't
be processed by ACR 2.4, nor other Raw processors. Upgrades will be
needed.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/
McLeod - 28 May 2005 15:51 GMT
>This shows the power of DNG. You can write code to process files
>conforming to particular versions of DNG, and you can cater
[quoted text clipped - 9 lines]
>be processed by ACR 2.4, nor other Raw processors. Upgrades will be
>needed.

What I would like to see is more open source image editing systems.

Adobe Systems is making me very nervous with their attempt to
commoditze digital imagery.  Once they have got us all using their
image processing system what is to stop them from hitting us at every
turn for more money. (Like CS2 and ACR 3.1).  

From the Dave Coffin interview at dpreview
http://www.joelonsoftware.com/articles/StrategyLetterV.html
Barry Pearson - 28 May 2005 17:44 GMT
[snip]
> What I would like to see is more open source image editing systems.

So would I. I am especially interested in John Francis' "Raw-to-DNG
converter" SourceForge project, which will remove the one case where we
are currently dependent on a single supplier. I am looking forward to
there being a Converter that isn't constrained by Adobe's legal issues.

> Adobe Systems is making me very nervous with their attempt to
> commoditze digital imagery.  Once they have got us all using their
> image processing system what is to stop them from hitting us at every
> turn for more money. (Like CS2 and ACR 3.1).
[snip]

Photoshop (and other photo-editors) have been tending to commoditise
digital imagery since TIFF 6.0! It is nothing new. Everything that can
deliver a TIFF 6.0 file into Photoshop is having to compete with
everything else that can do it. But it can argued that, by outputting
in TIFF 6.0 format, the camera manufacturers have been tending to
commoditise photo-editors that accept TIFF 6.0 files, by forcing them
to compete with one another. It isn't just a one-sided activity!

Now Adobe Camera Raw (and other Raw processors) are tending to
commoditise products that output in Raw format. The latter are all
having to compete with one another. And, conversely, the camera
manufacturers are tending to commoditise Raw processors, of course.

But what Adobe are doing with DNG is very different from "Photoshop
versus camera manufacturers". Adobe are trying to open up the whole
industry and marketplace of Raw shooting and processing. Paradoxically,
opening up an industry starts to remove the "dominant supplier" bias.
If/when DNG becomes ubiquitous, people will be able to choose their Raw
processors and asset management systems purely on merit. And they will
also be able to choose their cameras purely on merit, instead of having
to be guided by the fact that the main camera manufacturers tend to own
the better software, and are better supported by Raw processors and
asset management systems.

In what way are Adobe "hitting us at every turn for more money. (Like
CS2 and ACR 3.1)"? If you don't want to upgrade, don't! Why will you
ever need to pay Adobe any more money? As shown here, if you buy a
camera that CS and ACR 2.4 doesn't support, use the free DNG Converter
to convert that camera's Raw files to DNG, and you will be able to
stick with CS and ACR 2.4. That route *really* does work! I've tried
it, and so have many others.

If/when DNG becomes ubiquitous, we will all be able to build our
digital photography system by choosing individual components, such as
cameras, Raw processors, photo-editors, etc, on their merits. We will
be able to do mix-n-match. Perhaps we will sometimes have alternatives
to use for different purposes. This is the sort of future for top-end
digital photography that all photographers, and users of photographs,
should be hoping for.

What will Adobe get out of all this? Simple! The more freedom people
have to choose, the more likely it is that a large proportion of them
will buy Adobe products. Because they are the highest value digital
imaging products around. THAT is why people will buy Photoshop; choice,
not force.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/
Barry Pearson - 28 May 2005 09:58 GMT
> > Frankly, I haven't a clue whether the "unique black level estimation
> > for long exposure images" is catered for by the current version of DNG.
>
> It isn't.

What evidence is there for this?

[snip]
> Nikon (or whoever) has likely has something that is being dropped on
> the floor when you convert to DNG.  This is the basic point (that is
> apparently lost on too many people):  without specific support for all
> features present in a camera, DNG is necessarily lossy.

I've just covered this in another response. What matters is whether DNG
has sufficient richness for all the details of the camera model needed
for processing the Raw file. It appears to me that it has for the D2X
and the 350D, because I could get good images via this route. But I
accept that this was just for a limited number of images. I didn't
process any that had a long exposure.

Had I tried such images, what problems would I have had? Is it
something that couldn't be handled even in theory by Raw processors
that didn't read the "unique black level estimation for long exposure
images"? Or simply something that has to be catered for another way?

Remember the fuss about D2X WB encryption. DNG files generated from D2X
Raw files don't (currently) properly identify the "as shot" WB, so in
that sense those *files* are lossy. And for a lot of photographers,
this turns out to be unimportant. But it doesn't mean that the DNG
*format* is inherently lossy; what is lossy is the conversion process
when performed by software that can't get at the WB. If the D2X camera
were to output DNG files, it might well choose to do the job properly,
and avoid loss. An open source DNG Converter, unconstrained by Adobe's
fears of legal action, might write more complete DNG files from the
D2X. I think we will see some interesting open source, and other
independent, applications of DNG over the next year or so.

[snip]
> The existence of "PrivateData" is the problem, not the lack or presence
> of version numbering.
[quoted text clipped - 4 lines]
> bother storing it at all?  In this case, DNG offers you nothing more
> than what a TIFF file would.

Not true. For example, DNGs generated from the Raw files of my camera
have data that isn't understood by standard DNG readers, and it makes
no significant difference to me. I can't get at the lens model. If I
could, it might be possible to write a script to analyse all the lens
information in the metadata and apply automatic chomatic abberation
reduction. But, in the absence of that, I do it by eye.

I think this is the likely use of the private data. To give the camera
manufacturer's software opportunities to do things an easy way that
other processors have to do a harder way. (Without offering such a
"unfair advantage" to the camera manufacturers, it would be far harder
to persuade them to use DNG. Perhaps it is a "necessary evil").

DNG gives me, and others who use it, the advantages of Raw processing
over TIFF processing.

> b) if some DNG reader _can_ understand it, it must be a custom reader,
> and thus one has just converted the "file format problem" into an
[quoted text clipped - 3 lines]
> situation?  In this case, DNG has given you nothing but an extra, lazy,
> man in the middle that drops bits on the floor.

Not true. See my example above. I agree that a formal standard for
archival purposes would be unlikely to have private data fields.
Proposals for an archival version of PDF, called PDF-A, don't have
encryption, for example. I can envisage an archival version of DNG,
perhaps DNG-A, that has no private bits.

> Note that in both cases the DNG (or OpenRAW or whatever) gives you
> nothing.  Why, then, use it?

I have been using DNG since 10th October 2004. It gives me benefits,
more with each version of ACR, more with each version of the DNG
Converter, and lots more with the upgrade to CS2.

I get much smaller files, which makes processing faster and archiving
easier. I get my ACR settings and adjustments stored in the DNG file,
so that I have just one file to deal with. I have the confidence that I
will be able to process those archived DNG files for as long ahead as I
need, and I didn't have this confidence with the files directly from my
camera.

I know that ACR gives the same image whether I process the camera's
original files or DNG files. (I have tested Raw files for a number of
cameras of a number of makes, and in each case the result of the ACR
processing was the same for both the original Raw file and the DNG
version). If I buy a new camera after CS3 and ACR 4.x have been
released, I will probably not have to upgrade to CS3 to handle that new
camera.

The 3.1 DNG Converter stores the private data of my camera in the DNG
file. Although I can't currently do anything with it, at least it is
there. Perhaps in future I will be able to do something with it.
Perhaps a new DNG Converter will accept my current DNG files and move
that data to a place where it can be used directly. Who knows?

I think in future we will see high quality magazines, and other users
of photographs, accepting Raw files. I'm confident they will accept
DNG. They may sometimes insist upon it, just as now they may insist
upon TIFF 6.0.

DNG is opening up the world of Raw processing. I think we will see
benefits in future that we can't yet think of.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/
Owamanga - 27 May 2005 19:35 GMT
>Now if the DNG can't handle that, what do you recommend I do with my
>multi-minute minute exposure CR2 files I have?

Dunno, but based on the original article, I'm just looking forward to
playing all my Betamax tapes on my VHS recorder again, which it almost
promises the DNG standard will be capable of doing.

Traci Lords anyone?

--
Owamanga!
http://www.pbase.com/owamanga
Barry Pearson - 27 May 2005 08:29 GMT
[snip]
> I can't comment on Nikon stuff, since I'm a Canon sort of person.
>
[quoted text clipped - 4 lines]
> managed to figure it out speaks for his abilities as a Reverse
> Engineer.
[snip]

By "weird TIFF", do you mean "weird TIFF 6.0" or "weird TIFF/EP"? Is
it, in fact, "standard TIFF/EP"? You may know the following, but I
suspect many people don't:

1. *TIFF 6.0* is owned by Adobe, and was owned by Aldus before that.

When you set your camera to TIFF, that really means TIFF 6.0. Raw
processors normally have TIFF 6.0 as an *output* option. Photo-editors
normally read and/or write TIFF 6.0 as options. (You can actually use
TIFF 6.0 instead of PSD for Photoshop - make sure you ask for layers to
be preserved). If a magazine buys one of your photographs in TIFF form,
they mean TIFF 6.0.

TIFF 6.0 images have the full colour information for each pixel,
(except for monochrome images, of course). TIFF 6.0 is not suitable for
Raw files, because it has no tags defined for holding sensor data, and
there are many other metadata tags that it would need also.

2. *TIFF/EP* ("Electronic Photography") is owned by ISO, and costs 150
euros. It was developed from TIFF 6.0, and much of it is copied
directly from TIFF 6.0.

It has those extra tags for sensor data, (look for tags starting CFA -
Color Filter Array). It also has a lot of extra metadata tags. Camera
manufacturers typically (but not necessarily) start with TIFF/EP when
designing their Raw formats, because it has many of the tags they need.
Indeed, one manufacturer originally used the .TIF extension for its Raw
files.

Like TIFF 6.0, TIFF/EP has very many options. And, of course, although
it is a standard, there is no constraint on manufacturers to conform
precisely to it. So resultant Raw formats proliferate, and there are
lots of unnecessary differences.

3. *DNG*, like TIFF/EP, is a development of TIFF 6.0, but I prefer to
think of it as a development of TIFF/EP. It has a few tags that TIFF/EP
doesn't have, to cater for technology innovations since TIFF/EP was
defined, and for control purposes, such as the 2 version tags that
define the range of DNG versions that a particular file is compatible
with.

DNG cuts through all the unnecessary options in TIFF/EP, eliminates
some of the more complicated structural options, and makes some tags
mandatory. So it ensures that the image data can always be extracted,
along with sufficient extra information that Raw processors can handle
the image data without needing to know about the camera. (The file is
self-contained). But it also defines how camera manufacturers can
safely store any "secret sauce" in a DNGPrivateData tag defined for the
purpose. "Safely" means that anyone else can re-write a DNG file while
preserving the private data, because although its contents are private,
its format is well-structured.

4. To summarise: TIFF 6.0 is what people tend to mean when they say
"TIFF" without qualification. TIFF/EP is TIFF 6.0 plus a lot of the
stuff needed to make Raw files. DNG is TIFF/EP brought up to date and
made fit for purpose. (Now I'll duck for cover!)

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/
eawckyegcy@yahoo.com - 27 May 2005 17:16 GMT
> By "weird TIFF", do you mean "weird TIFF 6.0" or "weird TIFF/EP"?

I mean "weird TIFF", as "unusual", "not exactly the same as but similar
to in strange way", or "derived from by those under the influence of
unspecified narcotic agents", and so forth.  It has a directories,
tags, and the rest of it.  Everything else is completely incompatible
with TIFF as is, but it can be easily converted into whatever TIFF one
wishes (if anyone was so inclined).

> But it also defines how camera manufacturers can safely store any "secret
> sauce" in a DNGPrivateData tag defined for the purpose. "Safely" means
> that anyone else can re-write a DNG file while preserving the private
> data, because although its contents are private, its format is
> well-structured.

You need to read the ranting of the OpenRAW people.  They explicitly
demand "no secret sauce" -- though irrationally maintaining they are
not making such a demand.
Ben Rosengart - 27 May 2005 19:35 GMT
> 4. To summarise: TIFF 6.0 is what people tend to mean when they say
> "TIFF" without qualification. TIFF/EP is TIFF 6.0 plus a lot of the
> stuff needed to make Raw files. DNG is TIFF/EP brought up to date and
> made fit for purpose. (Now I'll duck for cover!)

Thanks for this explanation.  I found it quite enlightening.

Signature

Ben Rosengart                                            (212) 741-4400 x215
    Sometimes it only makes sense to focus our attention on those
    questions that are equal parts trivial and intriguing.
                                            --Josh Micah Marshall

Ryadia@Home - 26 May 2005 21:36 GMT
>>Open RAW is best for everyone in the long term.
>
> Be very careful what you wish for.

Yes. Mrs Browne wished for a boy with good looks, smart wit and high
intellect. Unfortunately she got Alan!.

Douglas
Alan Browne - 26 May 2005 21:45 GMT
> Yes. Mrs Browne wished for a boy with good looks, smart wit and high
> intellect. Unfortunately she got Alan!.

Your mother thought anal sex with a donkey wouldn't get her pregnant.

Signature

-- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm
--        r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm
--      [SI] gallery & rulz: http://www.pbase.com/shootin
--                   e-meil: Remove FreeLunch.

Barry Pearson - 27 May 2005 01:02 GMT
> > Open RAW is best for everyone in the long term.
>
[quoted text clipped - 3 lines]
> Offer praises to Allah or whoever that cell phones, VOIP and the rest
> of it are doing the job indirectly.

Perhaps, without that standard, we would not be advanced enough to make
that wish! We would still be wasting too much time trying to get basic
communications to work.

We sort out one layer of the communications interworking. That enables
us to build lots of services in higher layers. That creates a desire
for better lower layers. The trick is to have an architecture, or
structural decomposition, that enable to make progress in one area
without having to tear up all other areas. And that needs clean
interfaces between the various areas.

If we DO decide to rip all that stuff out, I would hope we could do so
without (say) having to re-invent TCP/IP to run over its successor.

> Be very careful what you wish for.

I would think that anyone who has spent many years or decades helping
to design complex multi-vendor systems knows what to wish for here! We
need a common Raw format. And in the meantime we need sufficient
publication of our current formats to build stop-gap measures such as
access to proprietary Raw formats, and conversion to a common Raw
format, etc.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/
Owamanga - 26 May 2005 22:55 GMT
>>http://www.luminous-landscape.com/essays/raw-flaw.shtml
>>
[quoted text clipped - 3 lines]
>any data in the image created by the camera you own to be obfuscated or
>encrypted.

With that I agree, my complaint is with stupidity such as this:

"What happens when your new Quatum Cube based computer no longer can
read CDs, or DVDs, or its operating system can't deal with something
as old and arcane as Windows XP or Mac OSX?

Far fetched you say? We'll, how many of you have a stack of 3.5"
floppies somewhere in your des