magenta highlights

false color highlights seem to be an issue frequently, so here’s some quick faq about it. alexandre, please excuse all the outward references ;)

why are my highlights magenta?

that’s how the sensor works. it collects a couple of photons, at some point it fills up and rejects to deliver any more useful information past this point. unfortunately that doesn’t happen at the same time for all color channels.

how does the sensor work?

usually digital cameras come with a color filter array (CFA) of absorptive filters in front of an array of CCD or CMOS sensor which collect electrons proportional to the incoming photons according to the photoelectric effect.

most cameras use some simple variation of an RGGB layout in 2x2 blocks for these color filters, and we must take care to reconstruct three color channels per pixel out of these one different color per pixel formats (demosaicing). some manufacturers choose to give their cameras more esoteric layouts.

so that’s why my foveon/fuji camera is not supported!

yes.

what about black/white digital cameras?

these don’t have a CFA and can thus collect more photons (none are absorbed) and don’t need demosaicing. should result in more sharpness and higher useful iso.

however, some of the rest here still applies to these models too, if you whitebalance to compensate for illumination.

back to highlights though

once a photon comes past the absorptive filter, it is converted to an electron and stored until it is read out to form the image. maybe your chip has a full well capacity of 90000 electrons. after that it overflows and still returns 90000 (it’s saturated, your value clips).

this is in turn converted to a value between 0 and 4096 (say, if your camera supports 12 bits. note that this is also the reason why setting your camera to high iso makes sense as compared to just cranking up exposure in post. you lose this precision on-chip). then you whitebalance your image. this is just a multiplication with a scalar value per channel, and meant to compensate both the different absorption in the color filters, and the colorful illumination. this can make a photograph of a grey patch look grey in your image again.

typically your whitebalance coefficients will be something like (2.0, 1.0, 1.4), normalized to 1.0 for green. this means that an overexposed picture of the sun will fill up all your sensors and result in a value of (2.0, 1.0, 1.4) in your final image, which is magenta.

wait, doesn’t that confuse demosaicing?

yes, at high contrast edges (mountain ridge against overexposed sky). this typically results in purple fringing.

what can we do about it?

the obvious thing to do is to clip all three values at the minimum clipping threshold (the channel that fills up quickest after white balancing dictates what value will give you fully saturated white). for instance whitebalance coefficients (2.0, 1.0, 1.4) will clip at 1.0 (green) and thus potentially waste half the photons collected for the red channel.

there’s also the old ufraw-style trick to `reconstruct’ some contrast in Lab color space. this works by converting the unclipped (false color) pixel to LCh, and discarding the saturation (C) and hue (h). actually we let you choose what to discard and by what percentage.

find these two options in the `highlight reconstruction’ plugin in the basic group.

how is hdr different

the clipping threshold is determined during hdr construction (merging a couple of exposures together), and bracketing might expose issues with a slightly off value much more gravely.

where is the clipping threshold

in theory it only depends on your sensor’s full well capacity. choose this value right once and you’re done. you would think. these values are in rawspeeds cameras.xml file for example, and very carefully chosen and most of time they’re right. it turns out that some of these might be conservative (clipping too late) to preserve more information. this is a good thing and will give you most data out of your camera. but apparently there are a couple of edge-case pictures that will show the magenta problem nevertheless.

this is especially an issue for some hdr shots and in case the basecurve is switched off (which will usually push values closer to white by making them all brighter). so, whatever, in current git you now have a slider for this value. keep in mind you should never need to touch this (highlight reconstruction/clipping threshold, basic group).

1. binoyte on Thu Jul 05 18:40:35 2012:
   thank you for this most interesting article.
2. tosca on Fri Jul 06 07:01:28 2012:
   Thank you for this well documented article, that is not very understable for the user (vs developer or geek or mathematician) I am.
   I'm looking for some solution, even basic, so I can get reasonably good-looking pictures. I might be very down-to-earth on the matter, but Darktable seems to be the only program that I know about that produces this result; with the sams pictures, I have no such problem with Digikam derawtiser, RawTherapee or UFRAW.
   So why Darktable does have this problem? And how can I solve it without going to another program?
3. jo on Fri Jul 06 09:52:51 2012:
   the interesting program to compare to would be rawstudio, since these guys use the same backend to load raw files.
   
   as this thing takes so many forms, i have no idea what exactly you're talking about this time (still the overexposed bit in the glasses?). but as mentioned in the post, current git has a last-resort slider for the cases where your white point is inappropriate for some reason or another, so you can fix it manually in the gui. will take some time to drip through to a packaged version.
4. binoyte on Fri Jul 06 15:44:19 2012:
   bibble and Corel AfterShotPro (v1.0.0) used to have such magenta results with highlights recovery.
5. tosca on Fri Jul 13 14:01:41 2012:
   I have quite a lot of pictures that have magenta highlights. I showed the one with glasses because the magenta patch is geometrically well defined and very easy to see; I have plenty of others whose sky is magenta/rose-tinted, some of them have been published in my gallery : http://www.marie-noelle-augendre.com/photos/picture?/4314 (several places in the sky), http://www.marie-noelle-augendre.com/photos/picture?/4233 (on the rightmost wall).
   
   Is there a procedure somewhere to get the last git version? I seem to remember that is has been moved somewhere, and I've no idea how to get it. I tried the nightly build too, but it doesn't seem to work with Fedora 17.
   
   Is the information on this page http://www.darktable.org/install/ still up-to-date?
6. jo on Mon Jul 16 03:03:57 2012:
   wow, that does look pretty grave. you definitely want that new slider and an updated whitepoint for your camera.
   
   the git install instructions are up to date, the tricky part is to install all the build dependencies (which we conveniently list for ubuntu, but not for fedora which i don't know too much about).
   
   i would try a
   
   `yum-builddep darktable'
   
   before the ./build.sh or manual build step. another common pitfall is the install directory, i believe the default for build.sh is /opt/darktable (which makes sense as you can just delete the whole directory and be done if you want to uninstall it later).
   
   let me know if you try this and run into any trouble.
7. tosca on Mon Jul 16 18:54:49 2012:
   The compile part seemed to have gone pretty well (I had the dependencies already installed from a previous compilation). I'm going to make some tests on the 'bad' pictures, and let you know what result I can achieve.
   Thanks.
8. stonie81 on Fri Jul 27 08:32:56 2012:
   After updating to version 1.0.5 all my photos get this purple touch too :-/
   Whitepoint seems to get "lost" for my Pentax K-5, if i choose the white balance manually the purple goes away.
   Installed the GIT Version, same behaviour.
   By the way, love darktable! :-)
9. stonie81 on Sat Jul 28 22:02:35 2012:
   OK did a git pull (28.07. 23:00 UTC +2) and now the purple is gone :-)
10. Christian on Sun Dec 09 22:37:30 2012:
    ISO amplifies the signal analogically? I always thought the sensor yielded the same value I can find in the RAW data and then the ISO setting multiplied that value by some factor.