Photo Forum / Digital Photography / DSLR Cameras / June 2007
D200 compressed RAW comparison.
|
|
Thread rating:  |
Paul Furman - 27 Jun 2007 20:43 GMT As you may know, the D200 gives the option to shoot uncompressed RAW files which I had not bothered with as the files are much bigger but I did a test and I've changed my mind: <http://www.edgehill.net/1/?SC=go.php&DIR=Misc/photography/2007-06-27-raw-compressed> "This is a test to see if highlight recovery is better with uncompressed Nikon D200 RAW NEF files. My conclusion is yes, there is an improvement. Interestingly the NEF file was MUCH larger (15,851KB-vs-8,597KB) but not that different when converted to DNG (7,976KB-vs-7,039KB) with jpegs actually slightly reversed (3,737KB-vs-3,759KB). I also did one of the jpegs at default contrast & saturation, both of the raw conversions were tuned down one stop & all contrast settings bottomed out for highlight recovery.Shot on a 105 f/2.8 VR macro at about 6 ft away to a wicker chair backlit by a bright window."
 Signature Paul Furman Photography http://www.edgehill.net/1 Bay Natives Nursery http://www.baynatives.com
frederick - 27 Jun 2007 21:37 GMT > As you may know, the D200 gives the option to shoot uncompressed RAW > files which I had not bothered with as the files are much bigger but I [quoted text clipped - 10 lines] > recovery.Shot on a 105 f/2.8 VR macro at about 6 ft away to a wicker > chair backlit by a bright window." There's a comparison here: http://regex.info/blog/photo-tech/nef-compression/ The author extracts raw and compressed raw from the same exposure. Your test looks like it compares separate exposures, the softness of the compressed raw sample isn't from raw compression. There's a lot of visible CA/fringing for an expensive lens there too, which puts the priority of the raw vs compressed *nef argument in context of faults which degrade images - at the bottom of the list IMO.
Paul Furman - 27 Jun 2007 22:08 GMT >> As you may know, the D200 gives the option to shoot uncompressed RAW >> files which I had not bothered with as the files are much bigger but I [quoted text clipped - 14 lines] > There's a comparison here: > http://regex.info/blog/photo-tech/nef-compression/ I don't see any difference in his examples because he didn't use the raw converter to massage highlight recovery possibilities out of the files. That was my point. My samples show clearly that there is added detail in the uncrompressed raw file if you work a little to get it. Highlight recovery is a big deal, that's where information is lost forever so it's very useful to be able to get some of that detail back. If you aren't making adjustments in the raw converter there is no point in it.
> The author extracts raw and compressed raw from the same exposure. How? He compressed an uncompressed raw file?
> Your test looks like it compares separate exposures, Manual mode, same shutter & aperture, just changed the camera settings.
> the softness of the > compressed raw sample isn't from raw compression. Yes it is. There are no other factors changing.
> There's a lot of > visible CA/fringing for an expensive lens there too, which puts the > priority of the raw vs compressed *nef argument in context of faults > which degrade images - at the bottom of the list IMO. Yes, under extreme lighting even the best lenses will show CA/fringing. These were shot at 1/80 second f/6.3 (overexposed by about a stop). By massaging the file in the raw converter, you can reduce that fringing gracefully on an uncompressed raw file: that's useful!
Actually, in some parts of the image, I saw the CA/fringing more visible on the uncompressed file since it captures what the lens transmits more accurately but it also gives you more power to control the final product.
If you aren't getting my point about massaging the file in the raw converter, look at this test: http://www.edgehill.net/1/?SC=go.php&DIR=Misc/photography/expose-right What that tells you is to overexpose as much as possible (without blowing highlights) then correct/darken the exposure in the raw converter. Yet another reason to protect those blown highlights... that's they key area for image quality so it's worthwhile maxing out the highlight detail capture. And the dng conversion apparently applies some lossless compression so the only big hit is with memory card useage.
 Signature Paul Furman Photography http://www.edgehill.net/1 Bay Natives Nursery http://www.baynatives.com
frederick - 27 Jun 2007 22:42 GMT >>> As you may know, the D200 gives the option to shoot uncompressed RAW >>> files which I had not bothered with as the files are much bigger but [quoted text clipped - 22 lines] > very useful to be able to get some of that detail back. If you aren't > making adjustments in the raw converter there is no point in it. Yes, extreme highlight detail is lost in raw compression, in theory and in practice.
>> The author extracts raw and compressed raw from the same exposure. > > How? He compressed an uncompressed raw file? Yes - using this technique - explained in the blog at the bottom of the page: Step 1: capture an uncompressed-NEF image of your choice
Step 2: while compression still turned off, perform an in-camera image overlay with the step-1 picture and another image (a blank frame being best).
Step 3: change camera settings, turning NEF compression on
Step 4: repeat step 2
You now have two copies of the same image — the results of steps 2 and 4 — that are pixel-for-pixel identical except for NEF compression.
>> Your test looks like it compares separate exposures, > [quoted text clipped - 3 lines] > > Yes it is. There are no other factors changing. I don't believe it, I'd like to see you repeat that with the uncompressed/compressed method above.
>> There's a lot of visible CA/fringing for an expensive lens there too, >> which puts the priority of the raw vs compressed *nef argument in [quoted text clipped - 4 lines] > massaging the file in the raw converter, you can reduce that fringing > gracefully on an uncompressed raw file: that's useful! You shouldn't need to IMO with a shot taken with an expensive macro lens though! (That's a dig at the 105VR - which has more CA than third party lenses costing half the price!)
> Actually, in some parts of the image, I saw the CA/fringing more visible > on the uncompressed file since it captures what the lens transmits more [quoted text clipped - 9 lines] > highlight detail capture. And the dng conversion apparently applies some > lossless compression so the only big hit is with memory card useage. IMO, it's a mistake to expose so close to the point of blowing highlights. The penalty of extra noise from correcting slightly underexposed images is much less than the cost of blown highlights (possibly slightly exacerbated by raw compression). Of course we should be exposing perfectly every time.
Paul Furman - 28 Jun 2007 01:00 GMT >>>> As you may know, the D200 gives the option to shoot uncompressed RAW >>>> files which I had not bothered with as the files are much bigger but [quoted text clipped - 45 lines] > You now have two copies of the same image — the results of steps 2 and 4 > — that are pixel-for-pixel identical except for NEF compression. OK it took me a while to understand that the merging with a blank photo applies NEF compression in camera to the same original. I can see why this is useful in his test because he does the layer overlay difference comparison so each pixel needs to line up precisely. But... it doesn't matter that much for my situation because I'm testing what can be massaged out of the raw file and clearly seen without exaggeration. But yes, it's possible I introduced a little camera shake in spite of the tripod. I continue to maintain that his test is kind of meaningless though if he's not forcing some highlight recovery adjustments. That's the whole point is to be able to force more out of the file, not that it would look any different at default conversion.
>>> Your test looks like it compares separate exposures, >> [quoted text clipped - 6 lines] > I don't believe it, I'd like to see you repeat that with the > uncompressed/compressed method above. Alright, alright I tried it that way and while the file sizes are slightly different I could not see any difference at all, so I'm changing my mind & turning raw compression back on. Frankly I couldn't figure out how he did the exaggerated overlay mask, normally I'd paste one image over the other & set the top layer to mode 'difference' then I might add curves or levels to exaggerate subtle differences but in this case there was zero of anything at all to be seen. Pure black. I don't know how he got what he got but it must be very very minor. Maybe he used a different raw converter, maybe my dng conversion messed it up. I'm using CS1 ACR. BTW I had to shoot the blank overlay twice in both modes, using an uncompressed blank produced an identical sized file down to the last KB. The blanks were shot with the lens cap at max 1/8000 second.
I'm going to try it my way again with mirror lockup, remote cable & extra care not more the framing while changing the menu setting...
OK you win, there is no discernable difference. Using my overlay method I could see a slight shift from punching the menus... my first test apparently had camera shake from not using a cable release. So yeah, I'm turning on raw compression. Maybe that other test showed some very minor difference but if it's not perceptible under my somewhat extreme testing conditions, it's not meaningful.
>>> There's a lot of visible CA/fringing for an expensive lens there >>> too, which puts the priority of the raw vs compressed *nef argument [quoted text clipped - 10 lines] > though! (That's a dig at the 105VR - which has more CA than third party > lenses costing half the price!) Let's see that comparison... I think no lens is immune to these effects under extreme conditions and cheaper lenses would perform poorer.
>> Actually, in some parts of the image, I saw the CA/fringing more >> visible on the uncompressed file since it captures what the lens [quoted text clipped - 17 lines] > (possibly slightly exacerbated by raw compression). Of course we should > be exposing perfectly every time. The shadows can be greatly improved by pushing the highlights. With low contrast scenes, huge improvements can be had. It depends on the photo though.
 Signature Paul Furman Photography http://www.edgehill.net/1 Bay Natives Nursery http://www.baynatives.com
frederick - 28 Jun 2007 02:01 GMT > OK you win, there is no discernable difference. Using my overlay method > I could see a slight shift from punching the menus... my first test > apparently had camera shake from not using a cable release. So yeah, I'm > turning on raw compression. Maybe that other test showed some very minor > difference but if it's not perceptible under my somewhat extreme testing > conditions, it's not meaningful. Yep - I reckon that Jeffry Friedl has it summed up pretty well. There's a minor difference that you'll never normally see, and probably no meaningful benefit in real life from uncompressed raw. To put it in context, there's a majority out there who only ever shoot jpg, and a loud minority of jpg zealots who argue that shooting raw is a waste of time and effort. I don't.
> Let's see that comparison... I think no lens is immune to these effects > under extreme conditions and cheaper lenses would perform poorer. That's a generalisation. Cross brand, the Sigma 105 has less CA and higher MTF across the frame (dx) than the Nikkor 105VR. Some slower cheaper lenses (Nikkor 18-70) probably benefit from less critical optical design, and show less CA than the more expensive (and otherwise better) lenses like the 17-55.
achilleaslazarides@yahoo.co.uk - 27 Jun 2007 22:54 GMT > > the softness of the > > compressed raw sample isn't from raw compression. > > Yes it is. There are no other factors changing. So your conclusion is that one of the effects of the raw compression is softening?
> If you aren't getting my point about massaging the file in the raw > converter, look at this test:http://www.edgehill.net/1/?SC=go.php&DIR=Misc/photography/expose-right [quoted text clipped - 4 lines] > highlight detail capture. And the dng conversion apparently applies some > lossless compression so the only big hit is with memory card useage. Actually this highlight recovering by the raw converters is done (at least in some converters) by using the channels that aren't blown out to guess the one that was; so it's possible (and, I found, happens) to get colour shifts in highlights. This occurs if one of the channels is blown; different converters do this (ie guess the colour) better or worse.
It's possible to load a custom white balance to the camera (and set saturation, contrast etc) so that the histograms (and highlight clipping warnings) show clipping very close to the point where the raw data actually clips (I checked this using a raw converter that lets me see the actual raw values).
achilleaslazarides@yahoo.co.uk - 27 Jun 2007 23:58 GMT On Jun 28, 1:54 am, achilleaslazari...@yahoo.co.uk wrote:
> > > the softness of the > > > compressed raw sample isn't from raw compression. [quoted text clipped - 3 lines] > So your conclusion is that one of the effects of the raw compression > is softening? OK, I got curious so did a little experiment. I took this photo http://www.pbase.com/al599/image/81277114 in uncompressed raw. Checking in raw magick (a raw converter), the red and green channels are both blown somewhere (on the upper part), but certainly most of the red and green raw values are above 215 (which seems to be the value above which raw compression happens, at least in the D70).
I then sent it to someone who has nikon capture, and he resaved it as compressed raw (went down to 7mb) and sent it back, as well as the converted version. Nothing was changed (except white balance, both set to tungsten). I then overlaid the two files as layers (in 16-bit mode) and used difference blending. Setting the whitepoint in photoshop at 2 (ie moved it from 255 to 2), I got this: http://www.pbase.com/al599/image/81277117 (it's full size, click on "original" if it's not already).
Finally, I took the converted tiffs and sharpened them heavily (to make slight differences more visible), and got this: http://www.pbase.com/al599/image/81277115 (uncompressed) and http://www.pbase.com/al599/image/81277116 (compressed).
I also checked with another converter (raw magick), same thing. Now I did not do any highlight recovery, but if the data isn't there, I don't see how using uncompressed or compressed raw can affect it (after all, the only thing that can happen to non-existent entities is for them to be brought to existence; this isn't going to happen, in this case).
Some remarks: a) Maybe ACR treats the raw files differently, and there is a bigger difference if using ACR b) Maybe nikon capture's conversion to compressed NEF uses a different method than the camera. c) maybe the highlight recovery algorithms of some converters change depending on whether it's compressed or not d) maybe what is described here http://www.majid.info/mylos/weblog/2004/05/02-1.html doesn't apply to the d200, and it does something else; this "else" may be complicated enough that 1) it compresses the file by 50% with almost no visible degradation, unless 2) we use the exposure compensation slider set to -1, in which case the image softens. e) other (or some combination of the above).
achilleaslazarides@yahoo.co.uk - 28 Jun 2007 00:08 GMT On Jun 28, 2:58 am, achilleaslazari...@yahoo.co.uk wrote:
> certainly most of the red and green raw values are above 215 (which 215 out of 4095, that is
Paul Furman - 28 Jun 2007 01:24 GMT > On Jun 28, 1:54 am, achilleaslazari...@yahoo.co.uk wrote: > [quoted text clipped - 5 lines] >>So your conclusion is that one of the effects of the raw compression >>is softening? I've changed my mind, see above. Did you see any difference in your tests?
> OK, I got curious so did a little experiment. I took this photo > http://www.pbase.com/al599/image/81277114 [quoted text clipped - 22 lines] > I also checked with another converter (raw magick), same thing. Now I > did not do any highlight recovery, My version of ACR doesn't have actual highlight recovery, I just turned down the exposure, contrast, etc. Note the extreme software & methods used in the other test mentioned:
http://regex.info/blog/photo-tech/nef-compression/
The original compressed and uncompressed NEFs were converted to 16-bit “pseudo PPM” files of raw, linear sensor data with no gamma and no whitebalance, using David Coffin's most excellent dcraw.
I wrote a program to compare the two PPMs and generate a new 16-bit PPM representing only the per-pixel sensor-level differences, either as a mask or as a “true differences” image.
The dark/mid/light versions were produced from each NEF (using dcraw) with different -b (brightness) arguments.
The various jpeg versions of these files were created from PPM files using ImageMagick's convert utility.
The exaggerated versions were created using the convert utility's -normalize argument.
> but if the data isn't there, I > don't see how using uncompressed or compressed raw can affect it > (after all, the only thing that can happen to non-existent entities is > for them to be brought to existence; this isn't going to happen, in > this case). Well I'm assuming this highlight recovery business is all about those last few high numbers where there is a little something to work with. It seemed a shame to throw out any of those critical data points but now I think it is truly inconsequential.
> Some remarks: > a) Maybe ACR treats the raw files differently, and there is a bigger [quoted text clipped - 10 lines] > compensation slider set to -1, in which case the image softens. > e) other (or some combination of the above).
 Signature Paul Furman Photography http://www.edgehill.net/1 Bay Natives Nursery http://www.baynatives.com
achilleaslazarides@yahoo.co.uk - 28 Jun 2007 01:35 GMT > achilleaslazari...@yahoo.co.uk wrote: > >>So your conclusion is that one of the effects of the raw compression > >>is softening? > > I've changed my mind, see above. Did you see any difference in your tests? I'd tested this before but got curious again because of your results. I saw no difference worth speaking of (see links in my other post), so I'll keep using compressed raw too. It doesn't make any visible difference as far as I can tell.
newsmb@plcom.net - 28 Jun 2007 01:38 GMT > As you may know, the D200 gives the option to shoot uncompressed RAW > files which I had not bothered with as the files are much bigger but I [quoted text clipped - 12 lines] > Paul Furman Photographyhttp://www.edgehill.net/1 > Bay Natives Nurseryhttp://www.baynatives.com Is the question here one of, "Are the RAW files compressed using a lossy or non-lossy algorithm?". If the answer is the latter, then the only penalty should be slower performance. Surely Nikon make this information available?
Paul Furman - 28 Jun 2007 02:17 GMT >>As you may know, the D200 gives the option to shoot uncompressed RAW >>files which I had not bothered with as the files are much bigger but I [quoted text clipped - 14 lines] > only penalty should be slower performance. Surely Nikon make this > information available? It is lossy compression but inconsequential. I don't know if it has any impact on performance, in-camera processing times versus in-camera write times... it certainly has a big impact on file size and the time you wait to download off the card & storage space.
 Signature Paul Furman Photography http://www.edgehill.net/1 Bay Natives Nursery http://www.baynatives.com
frederick - 28 Jun 2007 02:25 GMT > Is the question here one of, "Are the RAW files compressed using a > lossy or non-lossy algorithm?". If the answer is the latter, then the > only penalty should be slower performance. Surely Nikon make this > information available? Nikon refer to compressed raw as "Visually Lossless". I think the links in this thread show that for practical purposes, then that's not a bad description.
|
|
|