Hello rader, everyone
First, many thanks and appreciation for posting and maintaining the spreadsheet. Great job.
I've finally got some time to review it thoroughly.
Below are my inputs, hope you find them useful.
1. RGB from input XYZ calculation, using the matrix specified in F56..H58.
The values specified are of the matrix used for transforming XYZ->RGB of the sRGB color space (not Rec. 709 RGB as specified in the above comment). Hence, caculated RGB values (actual: I56..K65, ideal: J72..L81) are inadequate for RGB Error calculations, and for estimating per-component gammas.
Furthermore, even if the correct Rec709 XYZ->RGB matrix was used, it applies only to HD signals, as SDTV (480i and 576i) use different RGB color spaces (different primaries), hence have different XYZ->RGB matrices (SMPTE RP 145 for 480i and EBU 3213 for 576i).
The user must be able to choose the standard being calibrated - either 480i, 576i or HD - and the matrix values should change accordingly. This assures the calculated RGB values correspond to the correct color space.
2. Gamma estimation
Terminology for the following section:
gamma: exponent _approximation_ (power function approximation: y = x^gamma)
gammaE: gamma used at encoding
gammaD: gamma at decoding. CRTs have intrinsic 2.5 gammaD (due to its built-in relationship betweem voltage input and light output in the electron gun)
transfer function: an explicit formula which defines the correlation between linear-light component and the gamma corrected video signal
When color is encoded (camera side), the Rec. 709 transfer function is used:
V' = 4.5L ; 0 <= L < 0.018
1.099(L^0.45) - 0.099 ; 0.018 <= L <= 1
(lets disregard the linear part for the following discussion)
Due to the scale factor and offset, stating Rec709 has a gammaE of 0.45 is wrong. When approximating this function to a pure power function, you'll get gammaE of about 0.52.
Futhermore, the inverse transformation, L = ((V' + 0.099) / 1.099) ^ 2.2222 simply describes how to recover the scene's linear light RGB values, NOT the RGB values produced by the display's transfer function - which is ofcourse unknown and _should not_ resemble rec709 inverse transformation.
For example, if a CRT is used, then its intrinsic L = V' ^ 2.5 is applied, resulting in an overall (end to end) gamma of 1.3 - suitable for dim environment watching.
I think we should assume display devices try to mimic CRT's intrinsic 2.5 gammaD, hence assume their transfer function is of the type L = (V' + e) ^ gammaD, where 'e' is an error introduced by the zero signal not truely displayed as black.
How this applies to the spreadsheet?
I think all gamma estimation caclulations, which assume display transfer function is similar to the inverse of Rec709 transfer function, are incorrect.
See V56, W56, P72, Q72, R72, S72, T72, U72.
I think that the "+0.099)/1.099" (offset and factor adjustments) should not be used.
GammaD should be estimated using the simple "L=(V'+e)^gammaD" model.
Since you already deduct black level error when normalizing Y values (see P56, formula is deducting Ymin value for normalization purposes) - you are left with even simplier model: L = V' ^ gammaD.
(and thanks to Poynton's excellent "Digital Video and HDTV" ...)
3. % RGB error
Error is calculated as the relative difference between actual RGB values (I56..K65) and the "ideal" RGB values (J72..L81).
However, the "ideal" RGB values are calculated using the _actual_ normalized Y, not the ideal normalized Y (see J72 which refers to D72,E72,F72 which all refer to P56).
Hence, we have RGB errors measured from the reference white point at the luminance level produced by the display (that has non ideal gamma).
An alternative approach might measure the RGB errors from the reference white point with an _ideal_ luminance for the input stimuli.
I'm not 100% sure, but I think that the current implementaion is indeed the desired one, however the issue is debatable...
4. Redundant calculations of ideal X Y Z (D72..F81) and ideal R G B (J72..L81).
Ideal X Y Z are calculated in order to derive ideal RGB values.
However, in any color system, _by definition_, the <1,1,1> RGB triple represents the system's reference white (with Y = 1), and any RGB triple whose components are equal, e.g , represent the reference white with Y = a.
Hence, ideal R G B values (J72..L81) must always be equal to one another, and their value must always be the same as Y (cell E72).
No need to calculate X and Z, no need for the XYZ->RGB matrix multiplication.
Again, many thanks.
Hope my comments make sense