How power law gamma calibration can lead to crushed blacks

I was encoding test patterns for a little project I'm working on and started digging into transfer functions and the linkage between gamma and saturation. It dawned on me that the current practice of calibrating one's display gamma to a simple power law with exponent 2.2-2.5 can lead to significant black crush as well as introducing unintended contrast into a scene. I did see a thread on this in the spectracal forums but not here so if it's been discussed before please forgive me.

ITU/EBU recommends that the end-to-end gamma of the video chain should be between 1.1 - 1.2 so we have a clearly defined target. It's commonly stated that video RGB is encoded with an exponent of 1/2.22 but the full function is:

v'=4.5*v for v < 0.018
v'=1.099*v^(.45)-0.099 for v >= 0.018

So what we want our displays to do is invert the function above and then apply a modest power of between 1.1 - 1.2.

What do we do in practice? We feed the display a linear input of 10% steps and (if we have a 10pt gamma control) try to get a nice flat fit to the equation v=v'^(gamma=2.2, 2.3, etc.) This completely ignores the fact that the real video signals we will eventually watch are encoded using the equation above.

So what are the consequences of the simplified power law calibration approach?



This is a plot of the result of feeding a linear signal which has been encoded using the BT.709/601 transfer function into a display with a simple power law response. Ideally we want a flat line somewhere within the cross-hatched region and you can see we don't get that. Most significantly the deviation below 20% stimulus can cause significant compression of the response as we go into black. You'll also notice that the response is not particularly flat until the upper stimulus levels. The implications of this is that contrast is being added at many of the levels which is not in the original video information.

I have been calibrating my display to a display gamma=2.3 and I find that I end up raising brightness a bit by eye to counter this effect when watching some material. If a 10pt gamma control is available a better approach is to actually calibrate to the BT.709/601 transfer function and this is an option in some of the calibration packages. Because it's not a simple power law, the way it's implemented (at least in HCFR) leads to a non-intuitive target gamma value (discussion here)

If you want to give this approach a try in HCFR use the "Camera gamma with standard offset" fitting function and targets of:

camera gamma = 2.44 for an end-to-end gamma = 1.1
camera gamma = 2.66 for an end-to-end gamma = 1.2

You'll find that to flatten out this gamma curve will require raising the 10%, 20%, and 30% controls (in decreasing amounts) relative to your standard calibration. I've seen people mention favoring a lower (power law) gamma at the low end and this achieves a similar result, but it's a much better match to how the video signal is actually encoded.


Before/After pictures, notice the overemphasis on shadow and loss of detail in the dress in the power law calibration.

power law gamma=2.3 End-to-End gamma is variable from 1.15 to 1.5



camera (inverse BT.709 transfer function) gamma=2.65 End-to-End gamma is flat at 1.2

If you target 2.2 yes because that results in end-to-end of 1.0, but target 2.65 and you'll get to the "dim surround" recommendation. That may still be too bright for theatre conditions but you can always push it higher (2.8 will approximate a simple power law value of 2.5), the other main advantage which I don't mention above is you get much better linearity between mid-tones (40-75% stimulus) and darks in the 10-20% range than a power law will give you. This gives a more natural looking image by avoiding the addition of false contrast (any slope in end-to-end gamma will add contrast to the image which is not encoded in the original).


I don't understand what you mean by "camera gamma isn't the answer". It's the exact inverse of the BT.709 encoding function. You start there to linearize the encoded signal and then tack on the additional 1.1 - 1.2 power.

BT.1886 recommends two different EOTFs designed to match a CRT response. This seems to be in contradiction to the end-to-end recommendation of 1.1-1.2 from Hunt.

Well that's a somewhat unsatisfactory state of affairs. So color science recommends a slightly non-linear signal chain for accurate perceptual reproduction and instead we get a system which has variable non-linearity (increasingly so as we approach black) but that's ok because it's being mastered on displays with the same non-ideal response function. Do directors, colorists, etc. realize they are not seeing a perceptually accurate rendition of their camera/telecine transfer signals?* Why weren't encoding transfer functions designed to better match CRT's? That would have made the world a tidier place.



*my guess is yes they do and adjust their displays accordingly.

What about telecine transfer of already produced films? Does the digitization work flow include a step which corrects for the mismatch between encoding function and a 2.35 CRT?

Another problem with mismatched transfer functions is color accuracy when saturation < 100%. You'll notice in the photographs above that in the 2.35 power law response that not only are the shadows overemphasized but the skin tones have been shifted to red. This can be explained with the following example:

1. Take a light skin tone (color checker pattern 2) with x=0.3871 y=0.3538 and Y=34.95%

2. Calculate R'G'B' using the BT.709 transfer function and you'll get 193,137,117 (0-255)

3. Now calculate the display xyY using a 2.35 straight power law and you'll get x=.4071, y=.3579, Y=28.6%

The y coordinate moves very little but x moves 0.051 towards red and Y is too dark. dE94 for this case is 5.7!

input------->output

ok but what about telecine transfer, or live broadcasts? Are you saying there is someone standing in between every delivery path adjusting levels so that it will appear correct on a 2.35 CRT at the user end?


Also, I've only done the one A/B comparison using the Blu-ray of "Tree of Life", but that material is clearly improved using this approach (on my panel).

So, exactly what are the studio guys using? Your assumption is that they grade and remap gamma from BT.709 to 2.35 and you assume this for all content, regardless of original source (film, video, TV (live or recorded)). If this assumption is incorrect you will have the problem stated in post 1, simple math.

Do we have anyone reading who actually works in these fields and can comment? The only previous comment I could find was from a thread that brought up similar issues:


This is of course completely contradictory to your assumption, but I have no way to know if it is an accurate statement or not of the industry wide practice.

Doug Blackburn takes a practical approach to the topic in that same thread basically saying don't take the numbers as gospel and see what looks right to you, using reference material. Of course this will work for the reference material used, but assumes all other material followed the same gamma workflow as the material he uses to grade the end-user display.

He also states:


Which is consistent with the encode/decode mismatch stated in post #1.

If we assume as above that gamma is remapped to appear correct on a 2.2 display the end-to-end gamma becomes:



This situation would also be consistent with Doug's observation and would imply that an inverse BT.709 function would not be an appropriate calibration target. The grading/LUT remapping takes care of it in production.

Thanks for the comments John. I've read that piece a number of times and it's still not getting at the crux of the problem.


In the previous paragraph he states by way of example that the colorist will increase chroma gain (to match a 2.4 gamma display btw). So over some range of luminances colors will appear correct assuming you reproduce the colorist's gamma function. This still does not address the gamma mismatch, the colorist can't make the two functions agree with each over all luminances with chroma gain alone. And it's still unclear whether this is actually standard practice given Light Illusion's comments.

I think I'll just have a beer and watch some TeeVee.

Thanks for the comments, they are very informative. The document you referenced can be found here

Regarding analog film to digital transfers, are you saying the DCI specs are followed as normal practice in the transfer to DVD/BD or were you referring to digital theatre projection?

I don't get a power law response from that formula, I get:



Unless I screwed up the calculation this is far from an idealized 2.4 power law.

Edit: See attached paper, Figure 1 - trace 2. Very similar CRT transfer function to what's plotted above. So if the single parameter power law fit is a poor representation of the actual CRT transfer function in use in production houses, the concept of reproducing a flat line gamma at the user end is seriously flawed.

 

20065167580318.pdf 114.2333984375k . file

Not an easy question to answer, as you have seen it all depends on your assumption about the characteristics of the mastering monitor. If we assume that a mastering system like the one Lewis Saunders uses (which he refers to as "the most common package for film colour management") then you would try to reproduce the curvature in my previous post and then adjust it up or down given your viewing environment.

If we just calculate the contrast ratio the colorist sees in such a system vs. the same ratio for flat gamma displays you get this:



2.4 in this case is better than the other options at these stimuli (it's worse above 35% but that has a bigger influence on color accuracy)

If we calculate the perceived differences in the first 12 Munsell color checker patches between a BVM raw display and a flat gamma display (assuming the gamut periphery has been calibrated to BT.709 and we are viewing the two monitors in the same environment) we get this:
 

Color	2.4 dE94	2.2 dE94
dark skin	2.7	4.8
light skin	4.5	3
blue sky	3.9	3.3
foliage	3.4	1.8
blue flower	4.3	1.7
bluish green	4.2	2.4
orange	2.2	1.9
purplish blue	2.6	1.9
moderate red	2.7	1.5
purple	2.1	2.6
yellow green	4.5	3.1
orange yellow	4.5	2.6

Other than the two darkest colors, a 2.2 gamma will give better color reproduction. Btw the above errors for either power law response would be considered unacceptable by most calibrators but I don't know of anyone who actually color grades their calibrations measures  "inside the edges". Of course, without a reference or an outboard LUT there isn't a whole lot you can do other than to characterize this region.

However, if the mastering system is using LUTed displays that produce nice flat gamma responses then the current practice is fine but we have no idea what percentage of what type of material is mastered on one or the other systems. You could hedge your bets and average the two or come up with a more complicated tailored function that minimizes the errors in the above table.

Edited by zoyd - 6/21/12 at 2:26pm

This is more of a preference question than a calibration question so I can't give you specific numbers. If you don't care to reproduce "what the colorist/director sees" then to increase shadow detail you lower gamma from 10%-30% and to increase contrast/saturation ("pop") you increase gamma from 30%-90% with a smooth transition between the two regions. Some displays have a dynamic contrast control which will do something similar but you'd have to decide for yourself what numbers to end up at.

Define most? "these days" is fine going forward but what of legacy material mastered using raw CRT EOTFs? the one industry person who has commented so far claims that "the most common package for film colour management" is such a system.