Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
PhotoKB Home
Discussion Groups
Digital Photography
Digital PhotoDSLR CamerasZLR CamerasPoint & Shoot Cameras
Film Photography
35 mmLarge FormatMedium formatDarkroomFilm and LabsOther Equipment
Photo Technique
Nature PhotographyPeople PhotographyTechnique General
General Photo Topics
General TopicsAustralian PhotographyUK Photography
DirectoryPhoto Clubs

Photo Forum / Digital Photography / DSLR Cameras / June 2007

Tip: Looking for answers? Try searching our database.

D200 compressed RAW comparison.

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
Paul Furman - 27 Jun 2007 20:43 GMT
As you may know, the D200 gives the option to shoot uncompressed RAW
files which I had not bothered with as the files are much bigger but I
did a test and I've changed my mind:
<http://www.edgehill.net/1/?SC=go.php&DIR=Misc/photography/2007-06-27-raw-compressed>
"This is a test to see if highlight recovery is better with uncompressed
Nikon D200 RAW NEF files. My conclusion is yes, there is an improvement.
Interestingly the NEF file was MUCH larger (15,851KB-vs-8,597KB) but not
that different when converted to DNG (7,976KB-vs-7,039KB) with jpegs
actually slightly reversed (3,737KB-vs-3,759KB). I also did one of the
jpegs at default contrast & saturation, both of the raw conversions were
tuned down one stop & all contrast settings bottomed out for highlight
recovery.Shot on a 105 f/2.8 VR macro at about 6 ft away to a wicker
chair backlit by a bright window."
Signature

Paul Furman Photography
http://www.edgehill.net/1
Bay Natives Nursery
http://www.baynatives.com

frederick - 27 Jun 2007 21:37 GMT
> As you may know, the D200 gives the option to shoot uncompressed RAW
> files which I had not bothered with as the files are much bigger but I
[quoted text clipped - 10 lines]
> recovery.Shot on a 105 f/2.8 VR macro at about 6 ft away to a wicker
> chair backlit by a bright window."

There's a comparison here:
http://regex.info/blog/photo-tech/nef-compression/
The author extracts raw and compressed raw from the same exposure.  Your
test looks like it compares separate exposures, the softness of the
compressed raw sample isn't from raw compression.  There's a lot of
visible CA/fringing for an expensive lens there too, which puts the
priority of the raw vs compressed *nef argument in context of faults
which degrade images - at the bottom of the list IMO.
Paul Furman - 27 Jun 2007 22:08 GMT
>> As you may know, the D200 gives the option to shoot uncompressed RAW
>> files which I had not bothered with as the files are much bigger but I
[quoted text clipped - 14 lines]
> There's a comparison here:
> http://regex.info/blog/photo-tech/nef-compression/

I don't see any difference in his examples because he didn't use the raw
converter to massage highlight recovery possibilities out of the files.
That was my point. My samples show clearly that there is added detail in
the uncrompressed raw file if you work a little to get it. Highlight
recovery is a big deal, that's where information is lost forever so it's
very useful to be able to get some of that detail back. If you aren't
making adjustments in the raw converter there is no point in it.

> The author extracts raw and compressed raw from the same exposure.

How? He compressed an uncompressed raw file?

> Your test looks like it compares separate exposures,

Manual mode, same shutter & aperture, just changed the camera settings.

> the softness of the
> compressed raw sample isn't from raw compression.

Yes it is. There are no other factors changing.

>  There's a lot of
> visible CA/fringing for an expensive lens there too, which puts the
> priority of the raw vs compressed *nef argument in context of faults
> which degrade images - at the bottom of the list IMO.

Yes, under extreme lighting even the best lenses will show CA/fringing.
These were shot at 1/80 second f/6.3 (overexposed by about a stop). By
massaging the file in the raw converter, you can reduce that fringing
gracefully on an uncompressed raw file: that's useful!

Actually, in some parts of the image, I saw the CA/fringing more visible
on the uncompressed file since it captures what the lens transmits more
accurately but it also gives you more power to control the final product.

If you aren't getting my point about massaging the file in the raw
converter, look at this test:
http://www.edgehill.net/1/?SC=go.php&DIR=Misc/photography/expose-right
What that tells you is to overexpose as much as possible (without
blowing highlights) then correct/darken the exposure in the raw
converter. Yet another reason to protect those blown highlights...
that's they key area for image quality so it's worthwhile maxing out the
highlight detail capture. And the dng conversion apparently applies some
lossless compression so the only big hit is with memory card useage.

Signature

Paul Furman Photography
http://www.edgehill.net/1
Bay Natives Nursery
http://www.baynatives.com

frederick - 27 Jun 2007 22:42 GMT
>>> As you may know, the D200 gives the option to shoot uncompressed RAW
>>> files which I had not bothered with as the files are much bigger but
[quoted text clipped - 22 lines]
> very useful to be able to get some of that detail back. If you aren't
> making adjustments in the raw converter there is no point in it.

Yes, extreme highlight detail is lost in raw compression, in theory and
in practice.

>> The author extracts raw and compressed raw from the same exposure.
>
> How? He compressed an uncompressed raw file?

Yes - using this technique - explained in the blog at the bottom of the
page:
Step 1: capture an uncompressed-NEF image of your choice

Step 2: while compression still turned off, perform an in-camera image
overlay with the step-1 picture and another image (a blank frame being
best).

Step 3: change camera settings, turning NEF compression on

Step 4: repeat step 2

You now have two copies of the same image — the results of steps 2 and 4
— that are pixel-for-pixel identical except for NEF compression.

>> Your test looks like it compares separate exposures,
>
[quoted text clipped - 3 lines]
>
> Yes it is. There are no other factors changing.
I don't believe it, I'd like to see you repeat that with the
uncompressed/compressed method above.

>>  There's a lot of visible CA/fringing for an expensive lens there too,
>> which puts the priority of the raw vs compressed *nef argument in
[quoted text clipped - 4 lines]
> massaging the file in the raw converter, you can reduce that fringing
> gracefully on an uncompressed raw file: that's useful!

You shouldn't need to IMO with a shot taken with an expensive macro lens
though!  (That's a dig at the 105VR - which has more CA than third party
lenses costing half the price!)

> Actually, in some parts of the image, I saw the CA/fringing more visible
> on the uncompressed file since it captures what the lens transmits more
[quoted text clipped - 9 lines]
> highlight detail capture. And the dng conversion apparently applies some
> lossless compression so the only big hit is with memory card useage.

IMO, it's a mistake to expose so close to the point of blowing
highlights.  The penalty of extra noise from correcting slightly
underexposed images is much less than the cost of blown highlights
(possibly slightly exacerbated by raw compression).  Of course we should
be exposing perfectly every time.
Paul Furman - 28 Jun 2007 01:00 GMT
>>>> As you may know, the D200 gives the option to shoot uncompressed RAW
>>>> files which I had not bothered with as the files are much bigger but
[quoted text clipped - 45 lines]
> You now have two copies of the same image — the results of steps 2 and 4
> — that are pixel-for-pixel identical except for NEF compression.

OK it took me a while to understand that the merging with a blank photo
applies NEF compression in camera to the same original. I can see why
this is useful in his test because he does the layer overlay difference
comparison so each pixel needs to line up precisely. But... it doesn't
matter that much for my situation because I'm testing what can be
massaged out of the raw file and clearly seen without exaggeration. But
yes, it's possible I introduced a little camera shake in spite of the
tripod. I continue to maintain that his test is kind of meaningless
though if he's not forcing some highlight recovery adjustments. That's
the whole point is to be able to force more out of the file, not that it
would look any different at default conversion.

>>> Your test looks like it compares separate exposures,
>>
[quoted text clipped - 6 lines]
> I don't believe it, I'd like to see you repeat that with the
> uncompressed/compressed method above.

Alright, alright I tried it that way and while the file sizes are
slightly different I could not see any difference at all, so I'm
changing my mind & turning raw compression back on. Frankly I couldn't
figure out how he did the exaggerated overlay mask, normally I'd paste
one image over the other & set the top layer to mode 'difference' then I
might add curves or levels to exaggerate subtle differences but in this
case there was zero of anything at all to be seen. Pure black. I don't
know how he got what he got but it must be very very minor. Maybe he
used a different raw converter, maybe my dng conversion messed it up.
I'm using CS1 ACR. BTW I had to shoot the blank overlay twice in both
modes, using an uncompressed blank produced an identical sized file down
to the last KB. The blanks were shot with the lens cap at max 1/8000 second.

I'm going to try it my way again with mirror lockup, remote cable &
extra care not more the framing while changing the menu setting...

OK you win, there is no discernable difference. Using my overlay method
I could see a slight shift from punching the menus... my first test
apparently had camera shake from not using a cable release. So yeah, I'm
turning on raw compression. Maybe that other test showed some very minor
difference but if it's not perceptible under my somewhat extreme testing
conditions, it's not meaningful.

>>>  There's a lot of visible CA/fringing for an expensive lens there
>>> too, which puts the priority of the raw vs compressed *nef argument
[quoted text clipped - 10 lines]
> though!  (That's a dig at the 105VR - which has more CA than third party
> lenses costing half the price!)

Let's see that comparison... I think no lens is immune to these effects
under extreme conditions and cheaper lenses would perform poorer.

>> Actually, in some parts of the image, I saw the CA/fringing more
>> visible on the uncompressed file since it captures what the lens
[quoted text clipped - 17 lines]
> (possibly slightly exacerbated by raw compression).  Of course we should
> be exposing perfectly every time.

The shadows can be greatly improved by pushing the highlights. With low
contrast scenes, huge improvements can be had. It depends on the photo
though.

Signature

Paul Furman Photography
http://www.edgehill.net/1
Bay Natives Nursery
http://www.baynatives.com

frederick - 28 Jun 2007 02:01 GMT
> OK you win, there is no discernable difference. Using my overlay method
> I could see a slight shift from punching the menus... my first test
> apparently had camera shake from not using a cable release. So yeah, I'm
> turning on raw compression. Maybe that other test showed some very minor
> difference but if it's not perceptible under my somewhat extreme testing
> conditions, it's not meaningful.

Yep - I reckon that Jeffry Friedl has it summed up pretty well.
There's a minor difference that you'll never normally see, and probably
no meaningful benefit in real life from uncompressed raw.  To put it in
context, there's a majority out there who only ever shoot jpg, and a
loud minority of jpg zealots who argue that shooting raw is a waste of
time and effort.  I don't.

> Let's see that comparison... I think no lens is immune to these effects
> under extreme conditions and cheaper lenses would perform poorer.

That's a generalisation.  Cross brand, the Sigma 105 has less CA and
higher MTF across the frame (dx) than the Nikkor 105VR.
Some slower cheaper lenses (Nikkor 18-70) probably benefit from less
critical optical design, and show less CA than the more expensive (and
otherwise better) lenses like the 17-55.
achilleaslazarides@yahoo.co.uk - 27 Jun 2007 22:54 GMT
> > the softness of the
> > compressed raw sample isn't from raw compression.
>
> Yes it is. There are no other factors changing.

So your conclusion is that one of the effects of the raw compression
is softening?

> If you aren't getting my point about massaging the file in the raw
> converter, look at this test:http://www.edgehill.net/1/?SC=go.php&DIR=Misc/photography/expose-right
[quoted text clipped - 4 lines]
> highlight detail capture. And the dng conversion apparently applies some
> lossless compression so the only big hit is with memory card useage.

Actually this highlight recovering by the raw converters is done (at
least in some converters) by using the channels that aren't blown out
to guess the one that was; so it's possible (and, I found, happens) to
get colour shifts in highlights. This occurs if one of the channels is
blown; different converters do this (ie guess the colour) better or
worse.

It's possible to load a custom white balance to the camera (and set
saturation, contrast etc) so that the histograms   (and highlight
clipping warnings) show clipping very close to the point where the raw
data actually clips (I checked this using a raw converter that lets me
see the actual raw values).
achilleaslazarides@yahoo.co.uk - 27 Jun 2007 23:58 GMT
On Jun 28, 1:54 am, achilleaslazari...@yahoo.co.uk wrote:

> > > the softness of the
> > > compressed raw sample isn't from raw compression.
[quoted text clipped - 3 lines]
> So your conclusion is that one of the effects of the raw compression
> is softening?

OK, I got curious so did a little experiment. I took this photo
http://www.pbase.com/al599/image/81277114
in uncompressed raw. Checking in raw magick (a raw converter), the red
and green channels are both blown somewhere (on the upper part), but
certainly most of the red and green raw values are above 215 (which
seems to be the value above which raw compression happens, at least in
the D70).

I then sent it to someone who has nikon capture, and he resaved it as
compressed raw (went down to 7mb) and sent it back, as well as the
converted version. Nothing was changed (except white balance, both set
to tungsten). I then overlaid the two files as layers (in 16-bit mode)
and used difference blending. Setting the whitepoint in photoshop at 2
(ie moved it from 255 to 2), I got this:
http://www.pbase.com/al599/image/81277117
(it's full size, click on "original" if it's not already).

Finally, I took the converted tiffs and sharpened them heavily (to
make slight differences more visible), and got this:
http://www.pbase.com/al599/image/81277115
(uncompressed) and
http://www.pbase.com/al599/image/81277116
(compressed).

I also checked with another converter (raw magick), same thing. Now I
did not do any highlight recovery, but if the data isn't there, I
don't see how using uncompressed or compressed raw can affect it
(after all, the only thing that can happen to non-existent entities is
for them to be brought to existence; this isn't going to happen, in
this case).

Some remarks:
a) Maybe ACR treats the raw files differently, and there is a bigger
difference if using ACR
b) Maybe nikon capture's conversion to compressed NEF uses a different
method than the camera.
c) maybe the highlight recovery algorithms of some converters change
depending on whether it's compressed or not
d) maybe what is described here
http://www.majid.info/mylos/weblog/2004/05/02-1.html
doesn't apply to the d200, and it does something else; this "else" may
be complicated enough that 1) it compresses the file by 50% with
almost no visible degradation, unless 2) we use the exposure
compensation slider set to -1, in which case the image softens.
e) other (or some combination of the above).
achilleaslazarides@yahoo.co.uk - 28 Jun 2007 00:08 GMT
On Jun 28, 2:58 am, achilleaslazari...@yahoo.co.uk wrote:
> certainly most of the red and green raw values are above 215 (which

215 out of 4095, that is
Paul Furman - 28 Jun 2007 01:24 GMT
> On Jun 28, 1:54 am, achilleaslazari...@yahoo.co.uk wrote:
>
[quoted text clipped - 5 lines]
>>So your conclusion is that one of the effects of the raw compression
>>is softening?

I've changed my mind, see above. Did you see any difference in your tests?

> OK, I got curious so did a little experiment. I took this photo
> http://www.pbase.com/al599/image/81277114
[quoted text clipped - 22 lines]
> I also checked with another converter (raw magick), same thing. Now I
> did not do any highlight recovery,

My version of ACR doesn't have actual highlight recovery, I just turned
down the exposure, contrast, etc. Note the extreme software & methods
used in the other test mentioned:

http://regex.info/blog/photo-tech/nef-compression/

The original compressed and uncompressed NEFs were converted to 16-bit
“pseudo PPM” files of raw, linear sensor data with no gamma and no
whitebalance, using David Coffin's most excellent dcraw.

I wrote a program to compare the two PPMs and generate a new 16-bit PPM
representing only the per-pixel sensor-level differences, either as a
mask or as a “true differences” image.

The dark/mid/light versions were produced from each NEF (using dcraw)
with different -b (brightness) arguments.

The various jpeg versions of these files were created from PPM files
using ImageMagick's convert utility.

The exaggerated versions were created using the convert utility's
-normalize argument.

> but if the data isn't there, I
> don't see how using uncompressed or compressed raw can affect it
> (after all, the only thing that can happen to non-existent entities is
> for them to be brought to existence; this isn't going to happen, in
> this case).

Well I'm assuming this highlight recovery business is all about those
last few high numbers where there is a little something to work with. It
seemed a shame to throw out any of those critical data points but now I
think it is truly inconsequential.

> Some remarks:
> a) Maybe ACR treats the raw files differently, and there is a bigger
[quoted text clipped - 10 lines]
> compensation slider set to -1, in which case the image softens.
> e) other (or some combination of the above).

Signature

Paul Furman Photography
http://www.edgehill.net/1
Bay Natives Nursery
http://www.baynatives.com

achilleaslazarides@yahoo.co.uk - 28 Jun 2007 01:35 GMT
> achilleaslazari...@yahoo.co.uk wrote:
> >>So your conclusion is that one of the effects of the raw compression
> >>is softening?
>
> I've changed my mind, see above. Did you see any difference in your tests?

I'd tested this before but got curious again because of your results.
I saw no difference worth speaking of (see links in my other post), so
I'll keep using compressed raw too. It doesn't make any visible
difference as far as I can tell.
newsmb@plcom.net - 28 Jun 2007 01:38 GMT
> As you may know, the D200 gives the option to shoot uncompressed RAW
> files which I had not bothered with as the files are much bigger but I
[quoted text clipped - 12 lines]
> Paul Furman Photographyhttp://www.edgehill.net/1
> Bay Natives Nurseryhttp://www.baynatives.com

Is the question here one of, "Are the RAW files compressed using a
lossy or non-lossy algorithm?". If the answer is the latter, then the
only penalty should be slower performance. Surely Nikon make this
information available?
Paul Furman - 28 Jun 2007 02:17 GMT
>>As you may know, the D200 gives the option to shoot uncompressed RAW
>>files which I had not bothered with as the files are much bigger but I
[quoted text clipped - 14 lines]
> only penalty should be slower performance. Surely Nikon make this
> information available?

It is lossy compression but inconsequential. I don't know if it has any
impact on performance, in-camera processing times versus in-camera write
times... it certainly has a big impact on file size and the time you
wait to download off the card & storage space.

Signature

Paul Furman Photography
http://www.edgehill.net/1
Bay Natives Nursery
http://www.baynatives.com

frederick - 28 Jun 2007 02:25 GMT
> Is the question here one of, "Are the RAW files compressed using a
> lossy or non-lossy algorithm?". If the answer is the latter, then the
> only penalty should be slower performance. Surely Nikon make this
> information available?

Nikon refer to compressed raw as "Visually Lossless".
I think the links in this thread show that for practical purposes, then
that's not a bad description.
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.