Thanks for this helpful discussion everyone!
Originally Posted by madshi
Ah, thanks Ted. It's good to see that AVSHD is the odd one out here. No idea why they would use truncating. I'd still like to know if HCFR takes rounding vs truncating vs dithering into account in its internal math? If the internal HCFR test pattern generator also uses rounding, but madTPG uses dithering, then does HCFR treat the measurement results differently/accordingly?
Thanks for your explanation on back calculating what AVSHD was doing. So let me just make sure I understand correctly:
1) The way most calibration tools display 16-235 grayscale windows is to round off to the closest integer to minimize the quantization error. This is consistent with the tables for Calman and Chromapure that connecteddd posted.
2) AVSHD & DVE overlooked something along the way because they are truncating/casting to integers which results in larger errors.
3) MadTPG doesn't truncate or round off...instead it uses dithering to achieve a mean level consistent with the floating point value corresponding to whatever percentage grayscale that is being displayed. Would this be correct?
Assuming I am right regarding all of the above, my follow-up question is:
1) Are tools like HCFR, Calman, Chromapure accounting for the fact that 40% grayscale (for example) is not really 40% grayscale due to the rounding off error, and in reality corresponds to 40.18% grayscale? If so, that makes sense and everything is as it should be.
2) I presume when HCFR is using madTPG, it treats 40% grayscale as 40% grayscale since madTPG is using dithering to achieve floating point RGB levels. Is this correct?
Originally Posted by zoyd
Yes, there is a switch to select either truncated or normal rounding for the gray scale calculations. With madTPG only FP equivalent of video level integers are used so if madTPG is set to video levels it won't dither, PC levels it will.
Thanks zoyd. I will look for this setting as I've never come across it previously. I am a bit confused about your 2nd sentence. Are you saying that if 0-255 is chosen in HCFR for the generator, then madTPG will use dithering to achieve floating point RGB levels, and that if 16-235 is chosen in HCFR, then dithering will not be used and madTPG will produce outputs with RGB levels consistent with the Calman/Chromapure table that ConnecteDDD posted earlier?
If I run a calibration with madTPG and another using the HCFR pattern generator for both 0-255 and 16-235, should I expect the HCFR and madTPG runs to be identical? Assuming that HCFR is keeping tally of which generator is rounding off as opposed to using dithering, I would think that both should provide near identical results?