HCFR - Open source projector and display calibration software - Page 138 - AVS Forum
Forum Jump: 
 64Likes
Reply
 
Thread Tools
post #4111 of 4431 Old 08-09-2014, 09:41 PM
Senior Member
 
orion2001's Avatar
 
Join Date: Jun 2008
Posts: 470
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 272 Post(s)
Liked: 159
Discrepancy between MadVR TPG and AVS 709 calibration

I'm having a bit of a tough time trying to figure out how to calibrate my TV that is hooked up to my HTPC (Intel HD4000 GPU). Initially, I calibrated using the HCFR internal pattern generator (view images) and got a great looking calibration. However, when I tried to verify the grayscale by playing the AVS 709 mp4 files via MPC-HC and MadVR, the results from 11 point grayscale measurements did not match those I obtained using the HCFR internal generator (Quite a big difference in avg gamma and gamma tracking).

I then decided that I could fix this by using madTPG. I got that working via HCFR and it seems that madTPG uses the same madVR settings as madVR in MPC-HC. I obtained a near perfect calibration to BT.1886 using madTPG and thought that this was the holy grail for me (since I play all video content via MPC-HC). However, when I played the mp4 files in MPC-HC, and tried to verify the grayscale...the results were again off from those obtained using madTPG. Keep in mind...the grayscale is still neutral and maintains very low dE values. However, the luminance tracking gets thrown off (thus the gamma is off) between my results from madTPG and manually playing AVS 709 mp4 files in MPC-HC via madVR.

Could anyone explain to me why this would be the case and what my options for calibration are? Am I relegated to calibrating the painful and tedious way of manually calling up each relevant mp4 file from AVS HD 709 files?
orion2001 is offline  
Sponsored Links
Advertisement
 
post #4112 of 4431 Old 08-10-2014, 05:04 AM
AVS Special Member
 
madshi's Avatar
 
Join Date: May 2005
Posts: 5,471
Mentioned: 6 Post(s)
Tagged: 0 Thread(s)
Quoted: 61 Post(s)
Liked: 129
Calibration DVDs and Blu-Rays are encoded in 8bit. Black is 16, White is 235. Now if HCFR wants to measure e.g. IRE 10, which 8bit value is that? It's not a cardinal number, so it's a matter of interpretation which exact 8bit value the DVD/Blu-Ray should be encoded to for e.g. IRE 10. Some DVDs may round, others may truncate. Some may even dither (as does madTPG).

Of course a good calibration should work for all content. So which test patterns should you use? Well, it doesn't matter that much, as long as the calibration software is aware of what it is measuring exactly. So I'm wondering: Does the AVS 709 calibration file round? Or does it truncate or dither? And does HCFR know and take into account how the AVS 709 file was encoded?

To be honest, I'm not sure if what I just wrote explains the discrepancy you're experiencing, but it's the first thing that came to my mind, so I thought I'd mention it.
madshi is online now  
post #4113 of 4431 Old 08-10-2014, 06:11 AM
Newbie
 
Join Date: Aug 2014
Posts: 2
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 0
Spyder4Pro Calibration Data

I'm confused about how to install the Calibration data for the Spyer4pro.

oeminst output-
Looking for 'C:\Program Files (x86)/Datacolor/Spyder4Pro/dccmtr.dll'
Loading file 'C:\Program Files (x86)/Datacolor/Spyder4Pro/dccmtr.dll'..done
...
oeminst: Error - Didn't locate any files to install - no CD present ?

Using existing meter correction file is blank. There's a note that says I have use the appropriate spectral sample correction available in the display type dropdown in the next dialog. There's no dropdown in the next dialog, just something about IR profiles?

After that there is a Display Type profile which I select LCD CCFL backlight. Is this all that's needed?
Matthew M. Dean (Culex) is offline  
post #4114 of 4431 Old 08-10-2014, 03:06 PM
Senior Member
 
orion2001's Avatar
 
Join Date: Jun 2008
Posts: 470
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 272 Post(s)
Liked: 159
Quote:
Originally Posted by madshi View Post
Calibration DVDs and Blu-Rays are encoded in 8bit. Black is 16, White is 235. Now if HCFR wants to measure e.g. IRE 10, which 8bit value is that? It's not a cardinal number, so it's a matter of interpretation which exact 8bit value the DVD/Blu-Ray should be encoded to for e.g. IRE 10. Some DVDs may round, others may truncate. Some may even dither (as does madTPG).

Of course a good calibration should work for all content. So which test patterns should you use? Well, it doesn't matter that much, as long as the calibration software is aware of what it is measuring exactly. So I'm wondering: Does the AVS 709 calibration file round? Or does it truncate or dither? And does HCFR know and take into account how the AVS 709 file was encoded?

To be honest, I'm not sure if what I just wrote explains the discrepancy you're experiencing, but it's the first thing that came to my mind, so I thought I'd mention it.
Hi madshi, thanks for taking the time to reply and for all your awesome work on madVR. I'm a bit ignorant when it comes to how files are encoded...are you saying that a file specifies whether dithering is to be applied to display a given level?

My understanding was that if the source is a DVD/BluRay with 16-235 levels, and if we were to then use madVR + MPC-HC to display this content on a TV that accepts 0-255, then the madVR settings would determine whether to use dithering or not, and how to round/truncate levels. Is this incorrect?

I was hoping that since madTPG seemed to be using the same madVR settings as in MPC-HC, that the decoding pipeline would be the same resulting in the same measurements. I do see your point though that madTPG may be determining slightly different level for a given IRE pattern compared to what is encoded in AVS 709 mp4 files.

If this is indeed the source of discrepancy, what would be the best way to go about calibrating? Should I stick to relying on madTPG results and would those be more in line with what I will see when using madVR+MPC-HC to playback bluray rips?
orion2001 is offline  
post #4115 of 4431 Old 08-10-2014, 09:11 PM
Advanced Member
 
gwgill's Avatar
 
Join Date: Jan 2013
Location: Melbourne, Australia
Posts: 633
Mentioned: 5 Post(s)
Tagged: 0 Thread(s)
Quoted: 76 Post(s)
Liked: 80
Quote:
Originally Posted by orion2001 View Post
Could anyone explain to me why this would be the case and what my options for calibration are? Am I relegated to calibrating the painful and tedious way of manually calling up each relevant mp4 file from AVS HD 709 files?
You are much better off digging in and figuring out what's going on, rather than guessing or asking other people to guess.

Do a screen capture of the two test patch sources, and check the captured values. That at least is a place to start...
gwgill is online now  
post #4116 of 4431 Old 08-10-2014, 10:36 PM
Senior Member
 
orion2001's Avatar
 
Join Date: Jun 2008
Posts: 470
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 272 Post(s)
Liked: 159
Quote:
Originally Posted by gwgill View Post
You are much better off digging in and figuring out what's going on, rather than guessing or asking other people to guess.

Do a screen capture of the two test patch sources, and check the captured values. That at least is a place to start...
Thanks Graeme! That was a great suggestion and I don't know why I didn't think to try that right away. After further investigations, it appears that the AVS709 mp4 files are the culprit. This is what I did:

1. Displayed 40% Gray patterns using madTPG and the corresponding mp4 file via MPC-HC and madVR. In both cases, output range was set to 0-255 in madVR settings for the TV. I took screenshots and saved PNG files and then opened them up in Photoshop.

2. In Photoshop, I noticed that there is some dithering in the colors for what should be uniform shade of grey. So I used Filter->Blur->Average to get an average color for a selection within the 40IRE window. This came out to 102,102,102 RGB for the madTPG window, but 101,101,101 for the AVS709 40% Gray window pattern displayed via MPC-HC and madVR.

So it seems that for whatever reason, AVS709 mp4 file is producing the wrong output as I believe that 102,102,102 should be the expected output for 40% gray windows.

I guess I should stick to using madTPG to calibrate my display with HCFR given these results? I've decided to use a custom 1-255 output range as it turns out that level 0 and level 1 are virtually identically displayed on the TV and results in level 1 essentially getting clipped out. Tweaking brightness values never helps with this. Using the 1-255 range sets blacks to level 1 and level 1 to RGB value of 2. This allows me to clearly distinguish all black levels above reference black at 16.
orion2001 is offline  
post #4117 of 4431 Old 08-10-2014, 11:12 PM
Advanced Member
 
gwgill's Avatar
 
Join Date: Jan 2013
Location: Melbourne, Australia
Posts: 633
Mentioned: 5 Post(s)
Tagged: 0 Thread(s)
Quoted: 76 Post(s)
Liked: 80
Quote:
Originally Posted by Matthew M. Dean (Culex) View Post
I'm confused about how to install the Calibration data for the Spyer4pro.

oeminst output-
Looking for 'C:\Program Files (x86)/Datacolor/Spyder4Pro/dccmtr.dll'
Loading file 'C:\Program Files (x86)/Datacolor/Spyder4Pro/dccmtr.dll'..done
...
oeminst: Error - Didn't locate any files to install - no CD present ?
You seem to have omitted the interesting bits - ie. what happened after loading dccmtr.dll, etc.

Try running oem inst with the CD present, or run oeminst on the setup.exe file on the CD, or the setup.exe from the DataColor website.
gwgill is online now  
post #4118 of 4431 Old 08-11-2014, 10:15 AM
AVS Special Member
 
madshi's Avatar
 
Join Date: May 2005
Posts: 5,471
Mentioned: 6 Post(s)
Tagged: 0 Thread(s)
Quoted: 61 Post(s)
Liked: 129
Quote:
Originally Posted by orion2001 View Post
are you saying that a file specifies whether dithering is to be applied to display a given level?
The file specifies which pixel has which color (usually defined in YCbCr 4:2:0).

Quote:
Originally Posted by orion2001 View Post
My understanding was that if the source is a DVD/BluRay with 16-235 levels, and if we were to then use madVR + MPC-HC to display this content on a TV that accepts 0-255, then the madVR settings would determine whether to use dithering or not, and how to round/truncate levels. Is this incorrect?
madVR never rounds/truncates, unless you explicitly tell it to do that. madVR *always* dithers by default. Which is the correct way of rendering video.

Quote:
Originally Posted by orion2001 View Post
I was hoping that since madTPG seemed to be using the same madVR settings as in MPC-HC, that the decoding pipeline would be the same resulting in the same measurements.
The processing pipeline is the same. It just so happens that most calibration files/discs do not really show the correct IRE value, but they round to the nearest 8bit value. (As I said in my previous comment, it's ok to do that as long as the calibration software is aware of that.)

Quote:
Originally Posted by orion2001 View Post
Should I stick to relying on madTPG results and would those be more in line with what I will see when using madVR+MPC-HC to playback bluray rips?
IMHO, yes.

Quote:
Originally Posted by orion2001 View Post
2. In Photoshop, I noticed that there is some dithering in the colors for what should be uniform shade of grey. So I used Filter->Blur->Average to get an average color for a selection within the 40IRE window.
Please note that averaging the color to 8bit (which is what Photoshop probably does) is very inexact. You say you're getting 102 with madVR and 101 with AVS709. This could be 102.499999 for madVR and 100.5 for AVS709. Or it could be 101.5 for madVR and 101.499999 for AVS709. So the difference between madVR and AVS709 could be anything between 0-2.
madshi is online now  
post #4119 of 4431 Old 08-11-2014, 04:38 PM
Advanced Member
 
gwgill's Avatar
 
Join Date: Jan 2013
Location: Melbourne, Australia
Posts: 633
Mentioned: 5 Post(s)
Tagged: 0 Thread(s)
Quoted: 76 Post(s)
Liked: 80
Quote:
Originally Posted by madshi View Post
Please note that averaging the color to 8bit (which is what Photoshop probably does) is very inexact.
You can avoid such rounding in PS by selecting the area and then using Histogram to show the floating point Mean for each channel.
gwgill is online now  
post #4120 of 4431 Old 08-11-2014, 05:03 PM
Senior Member
 
orion2001's Avatar
 
Join Date: Jun 2008
Posts: 470
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 272 Post(s)
Liked: 159
Quote:
Originally Posted by gwgill View Post
You can avoid such rounding in PS by selecting the area and then using Histogram to show the floating point Mean for each channel.
Neat! Thanks for that tip. I checked and my madTPG 40% pattern has a mean of 102 whereas the AVS709 mp4 file played in mpc-hc with madVR has a mean of 101.3. So it looks like the issue is with the mp4 file for whatever reason.

Thanks to everyone for all the help! I managed to achieve a really fantastic calibration on my PN60f5300 with HCFR and my colormunki display. As you can see, compromising on 100% saturation targets allowed me to achieve much better overall color tracking:



more calibration charts here: Samsung PN60F5300 calibration settings?
orion2001 is offline  
post #4121 of 4431 Old 08-12-2014, 01:20 AM
AVS Special Member
 
madshi's Avatar
 
Join Date: May 2005
Posts: 5,471
Mentioned: 6 Post(s)
Tagged: 0 Thread(s)
Quoted: 61 Post(s)
Liked: 129
Quote:
Originally Posted by gwgill View Post
You can avoid such rounding in PS by selecting the area and then using Histogram to show the floating point Mean for each channel.
Thanks!

Quote:
Originally Posted by orion2001 View Post
Neat! Thanks for that tip. I checked and my madTPG 40% pattern has a mean of 102 whereas the AVS709 mp4 file played in mpc-hc with madVR has a mean of 101.3. So it looks like the issue is with the mp4 file for whatever reason.
Ok, so let's do some math:

40% with 16-235 should be 16 + 219 * 0.4 = 103.6
40% with 0-255 should be 255 * 0.4 = 102.0

Calibration discs/files are almost always encoded in 16-235, so how did the AVS709 file encode the 103.6 value? It could truncate to 103, it could round to 104, or it could encode a pattern of 103 and 104 (dither) to achieve the exact 103.6 value.

So let's do the reverse math for AVS709 and we get 101.3 / 255 * 219 + 16 = 102.9988. So it seems that the AVS709 calibration file has 40% gray encoded as 16-235 8bit value 103. That looks like truncation to me which I find quite bad. They should at least have used rounding, then it should have been encoded as 16-235 8bit value 104, which displayed as 0-255 in madVR should average to about 102.5.

So my recommendation is to stay away from the AVS709 calibration file - unless you use a calibration software which is aware of that it has to deal with truncated test patterns.

@zoyd , would it make sense to add a feature to HCFR that allows the user to tell HCFR whether the (external) test patterns are truncated, rounded or dithered? That should allow HCFR to produce a more accurate report. Also, when using the internal test pattern generator (not madTPG), I suppose you're using rounding? Do you take the rounding into account when interpreting the measurement results in HCFR?
orion2001 likes this.
madshi is online now  
post #4122 of 4431 Old 08-12-2014, 02:47 AM
AVS Special Member
 
ConnecTEDDD's Avatar
 
Join Date: Sep 2010
Location: Athens, Greece
Posts: 2,330
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 97 Post(s)
Liked: 477
Quote:
Originally Posted by madshi View Post
So it seems that the AVS709 calibration file has 40% gray encoded as 16-235 8bit value 103.

So my recommendation is to stay away from the AVS709 calibration file - unless you use a calibration software which is aware of that it has to deal with truncated test patterns.
Hello madshi,

I have posted this chart at past; a quick example of Grayscale Patches RGB Triplet of AVSHD Disk vs. CalMAN/ChromaPure Engine's Targets is here:



HCFR is using the same RGB Triplet targets with ChromaPure for grayscale also.
orion2001 likes this.


To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts.

S/W: LightSpace CMS, SpaceMan ICC, SpaceMatch DCM, CalMAN 5, CalMAN RGB, ChromaPure, CalPC, ControlCAL
Meters: JETI Specbos 1211, Klein K-10A, i1PRO2, i1PRO, SpectraCAL C6, i1D3, C5
ConnecTEDDD is online now  
post #4123 of 4431 Old 08-12-2014, 02:53 AM
AVS Special Member
 
madshi's Avatar
 
Join Date: May 2005
Posts: 5,471
Mentioned: 6 Post(s)
Tagged: 0 Thread(s)
Quoted: 61 Post(s)
Liked: 129
Ah, thanks Ted. It's good to see that AVSHD is the odd one out here. No idea why they would use truncating. I'd still like to know if HCFR takes rounding vs truncating vs dithering into account in its internal math? If the internal HCFR test pattern generator also uses rounding, but madTPG uses dithering, then does HCFR treat the measurement results differently/accordingly?
ConnecTEDDD likes this.
madshi is online now  
post #4124 of 4431 Old 08-12-2014, 03:04 AM - Thread Starter
AVS Special Member
 
zoyd's Avatar
 
Join Date: Sep 2006
Location: Planet Dog
Posts: 4,619
Mentioned: 29 Post(s)
Tagged: 0 Thread(s)
Quoted: 194 Post(s)
Liked: 341
HCFR - Open source projector and display calibration software

Yes, there is a switch to select either truncated or normal rounding for the gray scale calculations. With madTPG only FP equivalent of video level integers are used so if madTPG is set to video levels it won't dither, PC levels it will.
ConnecTEDDD likes this.

-----

To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts.

Last edited by zoyd; 08-12-2014 at 03:10 AM.
zoyd is online now  
post #4125 of 4431 Old 08-12-2014, 03:17 AM
AVS Special Member
 
madshi's Avatar
 
Join Date: May 2005
Posts: 5,471
Mentioned: 6 Post(s)
Tagged: 0 Thread(s)
Quoted: 61 Post(s)
Liked: 129
Ah cool, thanks!

So, orion2001, you should get proper results with AVS709, too, if you switch HCFR to truncated gray scale calculations.
madshi is online now  
post #4126 of 4431 Old 08-12-2014, 03:20 AM
Member
 
ACappo's Avatar
 
Join Date: Jun 2013
Posts: 115
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 16 Post(s)
Liked: 43
avshd709 follows the rounding method applied by dve. why they did not round to neares integer, I do not know.
ConnecTEDDD likes this.
ACappo is offline  
post #4127 of 4431 Old 08-12-2014, 03:57 AM
AVS Special Member
 
madshi's Avatar
 
Join Date: May 2005
Posts: 5,471
Mentioned: 6 Post(s)
Tagged: 0 Thread(s)
Quoted: 61 Post(s)
Liked: 129
In the software development world there's usually a "round()" math instruction and a "trunc()" math instruction. Looking at Ted's table, avshd709 uses simple "trunc()", while CalMAN, ChromaPure, LightSpace, HCFR etc all use "round()". Yeah, ok, some people call what avshd709 does "round down". But in the software development world, this is simply called truncation and not rounding.

Point in case: IRE 5 calculates as 16 + 219 * 0.05 = 26.95. avshd709 uses "trunc(26.95) = 26". Everybody else uses "round(26.95) = 27". Basically with trunc(26.95) you get an error of 0.95. With round(26.95), the error is only 0.05. Which is why using truncation is usually a bad idea.
madshi is online now  
post #4128 of 4431 Old 08-12-2014, 05:36 AM
Advanced Member
 
gwgill's Avatar
 
Join Date: Jan 2013
Location: Melbourne, Australia
Posts: 633
Mentioned: 5 Post(s)
Tagged: 0 Thread(s)
Quoted: 76 Post(s)
Liked: 80
Quote:
Originally Posted by madshi View Post
In the software development world there's usually a "round()" math instruction and a "trunc()" math instruction. Looking at Ted's table, avshd709 uses simple "trunc()", while CalMAN, ChromaPure, LightSpace, HCFR etc all use "round()".
You are being generous. A more likely explanation is that they simply cast the calculation result to an integer without thinking about what that does.
gwgill is online now  
post #4129 of 4431 Old 08-12-2014, 06:39 AM
AVS Special Member
 
madshi's Avatar
 
Join Date: May 2005
Posts: 5,471
Mentioned: 6 Post(s)
Tagged: 0 Thread(s)
Quoted: 61 Post(s)
Liked: 129
You may be right... I forgot that in C++ you can simply type convert and you end up with truncation. In Delphi you explicitly have to use round() or trunc() or something similar to covert a floating point value to integer, otherwise you get a compiler error.
madshi is online now  
post #4130 of 4431 Old 08-12-2014, 08:47 AM
Member
 
ACappo's Avatar
 
Join Date: Jun 2013
Posts: 115
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 16 Post(s)
Liked: 43
Quote:
Originally Posted by ACappo View Post
avshd709 follows the rounding method applied by dve. why they did not round to neares integer, I do not know.
To clarify by dve i mean Digital Video Essentials
ACappo is offline  
post #4131 of 4431 Old 08-12-2014, 09:54 AM
Senior Member
 
carillon's Avatar
 
Join Date: Nov 2006
Location: Knoxville, TN
Posts: 207
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 17 Post(s)
Liked: 10
I'm using AVSHD and would like to enable the "truncating" setting within HCFR. Where can I find it? Thanks
carillon is offline  
post #4132 of 4431 Old 08-12-2014, 01:01 PM
Senior Member
 
orion2001's Avatar
 
Join Date: Jun 2008
Posts: 470
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 272 Post(s)
Liked: 159
Thanks for this helpful discussion everyone!

Quote:
Originally Posted by madshi View Post
Ah, thanks Ted. It's good to see that AVSHD is the odd one out here. No idea why they would use truncating. I'd still like to know if HCFR takes rounding vs truncating vs dithering into account in its internal math? If the internal HCFR test pattern generator also uses rounding, but madTPG uses dithering, then does HCFR treat the measurement results differently/accordingly?
Thanks for your explanation on back calculating what AVSHD was doing. So let me just make sure I understand correctly:

1) The way most calibration tools display 16-235 grayscale windows is to round off to the closest integer to minimize the quantization error. This is consistent with the tables for Calman and Chromapure that connecteddd posted.

2) AVSHD & DVE overlooked something along the way because they are truncating/casting to integers which results in larger errors.

3) MadTPG doesn't truncate or round off...instead it uses dithering to achieve a mean level consistent with the floating point value corresponding to whatever percentage grayscale that is being displayed. Would this be correct?

Assuming I am right regarding all of the above, my follow-up question is:

1) Are tools like HCFR, Calman, Chromapure accounting for the fact that 40% grayscale (for example) is not really 40% grayscale due to the rounding off error, and in reality corresponds to 40.18% grayscale? If so, that makes sense and everything is as it should be.

2) I presume when HCFR is using madTPG, it treats 40% grayscale as 40% grayscale since madTPG is using dithering to achieve floating point RGB levels. Is this correct?

Quote:
Originally Posted by zoyd View Post
Yes, there is a switch to select either truncated or normal rounding for the gray scale calculations. With madTPG only FP equivalent of video level integers are used so if madTPG is set to video levels it won't dither, PC levels it will.
Thanks zoyd. I will look for this setting as I've never come across it previously. I am a bit confused about your 2nd sentence. Are you saying that if 0-255 is chosen in HCFR for the generator, then madTPG will use dithering to achieve floating point RGB levels, and that if 16-235 is chosen in HCFR, then dithering will not be used and madTPG will produce outputs with RGB levels consistent with the Calman/Chromapure table that ConnecteDDD posted earlier?

If I run a calibration with madTPG and another using the HCFR pattern generator for both 0-255 and 16-235, should I expect the HCFR and madTPG runs to be identical? Assuming that HCFR is keeping tally of which generator is rounding off as opposed to using dithering, I would think that both should provide near identical results?

Last edited by orion2001; 08-12-2014 at 01:05 PM.
orion2001 is offline  
post #4133 of 4431 Old 08-12-2014, 02:42 PM
AVS Special Member
 
madshi's Avatar
 
Join Date: May 2005
Posts: 5,471
Mentioned: 6 Post(s)
Tagged: 0 Thread(s)
Quoted: 61 Post(s)
Liked: 129
Quote:
Originally Posted by orion2001 View Post
1) The way most calibration tools display 16-235 grayscale windows is to round off to the closest integer to minimize the quantization error. This is consistent with the tables for Calman and Chromapure that connecteddd posted.

2) AVSHD & DVE overlooked something along the way because they are truncating/casting to integers which results in larger errors.
Looks like that, yes.

Quote:
Originally Posted by orion2001 View Post
3) MadTPG doesn't truncate or round off...instead it uses dithering to achieve a mean level consistent with the floating point value corresponding to whatever percentage grayscale that is being displayed. Would this be correct?
madTPG isn't the one deciding which test pattern color should be shown. The calibration software which remote controls madTPG decides. madTPG shows whatever value is asked for.

Quote:
Originally Posted by orion2001 View Post
1) Are tools like HCFR, Calman, Chromapure accounting for the fact that 40% grayscale (for example) is not really 40% grayscale due to the rounding off error, and in reality corresponds to 40.18% grayscale? If so, that makes sense and everything is as it should be.
zoyd just wrote that there's an option for that in HCFR. Not sure about Calman and Chromapure. I guess they'll probably account for that, since their own test patterns work that way.

Quote:
Originally Posted by orion2001 View Post
2) I presume when HCFR is using madTPG, it treats 40% grayscale as 40% grayscale since madTPG is using dithering to achieve floating point RGB levels. Is this correct?
No, according to zoyd, HCFR is asking for rounded integer values which madTPG then displays as requested.

Quote:
Originally Posted by orion2001 View Post
If I run a calibration with madTPG and another using the HCFR pattern generator for both 0-255 and 16-235, should I expect the HCFR and madTPG runs to be identical? Assuming that HCFR is keeping tally of which generator is rounding off as opposed to using dithering, I would think that both should provide near identical results?
Why don't you test it, just to be safe, and then report your results here? I'd be interested in hearing your test results.
madshi is online now  
post #4134 of 4431 Old 08-12-2014, 04:06 PM - Thread Starter
AVS Special Member
 
zoyd's Avatar
 
Join Date: Sep 2006
Location: Planet Dog
Posts: 4,619
Mentioned: 29 Post(s)
Tagged: 0 Thread(s)
Quoted: 194 Post(s)
Liked: 341
Quote:
Originally Posted by orion2001 View Post
Thanks for this helpful discussion everyone!


Thanks for your explanation on back calculating what AVSHD was doing. So let me just make sure I understand correctly:

1) The way most calibration tools display 16-235 grayscale windows is to round off to the closest integer to minimize the quantization error. This is consistent with the tables for Calman and Chromapure that connecteddd posted.
Don't get hung up on % level patterns, there is no quantization error if the software knows which integers are being used.

Quote:
Originally Posted by orion2001 View Post
2) AVSHD & DVE overlooked something along the way because they are truncating/casting to integers which results in larger errors.
No, they purposely picked truncation to distribute 2 codes/% over a particular code range although it doesn't really matter. Again, if the software calculates the right target there is no error.

Quote:
Originally Posted by orion2001 View Post
3) MadTPG doesn't truncate or round off...instead it uses dithering to achieve a mean level consistent with the floating point value corresponding to whatever percentage grayscale that is being displayed. Would this be correct?
see madshi's response.

Quote:
Originally Posted by orion2001 View Post
Assuming I am right regarding all of the above, my follow-up question is:

1) Are tools like HCFR, Calman, Chromapure accounting for the fact that 40% grayscale (for example) is not really 40% grayscale due to the rounding off error, and in reality corresponds to 40.18% grayscale? If so, that makes sense and everything is as it should be.
HCFR is.

Quote:
Originally Posted by orion2001 View Post
2) I presume when HCFR is using madTPG, it treats 40% grayscale as 40% grayscale since madTPG is using dithering to achieve floating point RGB levels. Is this correct?
no, it either truncates or rounds nearest based on your preferences and sends the FP video level to madTPG.

Quote:
Originally Posted by orion2001 View Post
Thanks zoyd. I will look for this setting as I've never come across it previously. I am a bit confused about your 2nd sentence. Are you saying that if 0-255 is chosen in HCFR for the generator, then madTPG will use dithering to achieve floating point RGB levels, and that if 16-235 is chosen in HCFR, then dithering will not be used and madTPG will produce outputs with RGB levels consistent with the Calman/Chromapure table that ConnecteDDD posted earlier?

If I run a calibration with madTPG and another using the HCFR pattern generator for both 0-255 and 16-235, should I expect the HCFR and madTPG runs to be identical? Assuming that HCFR is keeping tally of which generator is rounding off as opposed to using dithering, I would think that both should provide near identical results?
the video vs. pc level setting has no effect on madTPG, HCFR always sends video level FP values to madTPG. For internal patterns HCFR sends either truncated or round nearest video or PC levels depending on your preferences and uses the correct levels to calculate targets.

-----

To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts.
zoyd is online now  
post #4135 of 4431 Old 08-12-2014, 09:50 PM
Senior Member
 
orion2001's Avatar
 
Join Date: Jun 2008
Posts: 470
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 272 Post(s)
Liked: 159
Quote:
Originally Posted by madshi View Post
Why don't you test it, just to be safe, and then report your results here? I'd be interested in hearing your test results.
Done!

Quote:
Originally Posted by zoyd View Post
Don't get hung up on % level patterns, there is no quantization error if the software knows which integers are being used.
Yup, makes sense.

So I did some tests comparing HCFR pattern generator, madTPG and AVS709. In the case of HCFR and madTPG, 0-255 levels was selected in the pattern generator options since my TV is setup for full range RGB levels. As zoyd mentioned, choosing 16-235 made no difference in madTPG results (pattern still showed as 0-255 due to madVR setting for monitor/TV as 0-255) but in HCFR, it resulted in washed out blacks (the BTB border around window patterns was visible).

madTPG and HCFR pattern generator results matched pretty well. It isn't a perfect match though, and I should point out that the variation seen is not purely down to meter variance between measurement runs. If I make multiple grayscale runs using a fixed pattern generator, the curves usually have much smaller variance... so this slight variance could be related in some small extent to small rounding/numerical errors or differences in implementation.

Dashed curve is madTPG grayscale measurement


Then, I selected the round-down option in HCFR for AVS709 and did a grayscale run using fullscreen AVS709 patterns played via MPC-HC and madVR. This is what I got...and I repeated again to get the same curve:



As you can see, even with round-down checked, there is some clear discrepancies. I took screenshots of AVS709 patterns and analyzed in Photoshop and the mean level was consistent with the round-down calculations as detailed by madshi in a couple of the previous posts. So it doesn't look like this is an issue related to playback chain on my HTPC. The AVS709 levels I am observing are consistent with round-down numbers (scaled to 0-255) as per ConnecteDDD's previous chart.

So, I'm not entirely sure why there is this discrepancy with AVS709. Might be something for someone else to look into in case this is a specific issue to me.
orion2001 is offline  
post #4136 of 4431 Old 08-13-2014, 05:58 PM - Thread Starter
AVS Special Member
 
zoyd's Avatar
 
Join Date: Sep 2006
Location: Planet Dog
Posts: 4,619
Mentioned: 29 Post(s)
Tagged: 0 Thread(s)
Quoted: 194 Post(s)
Liked: 341
Quote:
Originally Posted by orion2001 View Post
madTPG and HCFR pattern generator results matched pretty well. It isn't a perfect match though, and I should point out that the variation seen is not purely down to meter variance between measurement runs. If I make multiple grayscale runs using a fixed pattern generator, the curves usually have much smaller variance... so this slight variance could be related in some small extent to small rounding/numerical errors or differences in implementation.

Dashed curve is madTPG grayscale measurement
The differences you see here are because HCFR does not calculate the correct gamma when you select 0-255 PC levels output. For example, if you look at the 40% pattern in video levels this is 103.6 which gets rounded to 104 (40.1826%). 104 mapped to PC levels is 102.466. So madTPG will dither that to the correct luminance level but HCFR will round again to level 102 and be low by 0.466 bits (higher gamma). This would be ok if HCFR recalculated gamma when mapping to PC levels but it currently does not do that. If you look at the 50% level they agree better because the rounded 50% video level is 128.02 in PC levels so no dithering is needed. If you want to avoid this small error either use HCFR video levels end-to-end or use madTPG for PC levels.

Quote:
Originally Posted by orion2001 View Post
Then, I selected the round-down option in HCFR for AVS709 and did a grayscale run using fullscreen AVS709 patterns played via MPC-HC and madVR. This is what I got...and I repeated again to get the same curve:



As you can see, even with round-down checked, there is some clear discrepancies. I took screenshots of AVS709 patterns and analyzed in Photoshop and the mean level was consistent with the round-down calculations as detailed by madshi in a couple of the previous posts. So it doesn't look like this is an issue related to playback chain on my HTPC. The AVS709 levels I am observing are consistent with round-down numbers (scaled to 0-255) as per ConnecteDDD's previous chart.

So, I'm not entirely sure why there is this discrepancy with AVS709. Might be something for someone else to look into in case this is a specific issue to me.
To test the truncate switch you should compare AVSHD to another disk that uses round nearest, again with video levels end-to-end to avoid the 0-255 mapped rounding errors.

-----

To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 0 or greater. You currently have 0 posts.
zoyd is online now  
post #4137 of 4431 Old 08-13-2014, 07:13 PM
Senior Member
 
orion2001's Avatar
 
Join Date: Jun 2008
Posts: 470
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 272 Post(s)
Liked: 159
Quote:
Originally Posted by zoyd View Post
The differences you see here are because HCFR does not calculate the correct gamma when you select 0-255 PC levels output. For example, if you look at the 40% pattern in video levels this is 103.6 which gets rounded to 104 (40.1826%). 104 mapped to PC levels is 102.466. So madTPG will dither that to the correct luminance level but HCFR will round again to level 102 and be low by 0.466 bits (higher gamma). This would be ok if HCFR recalculated gamma when mapping to PC levels but it currently does not do that. If you look at the 50% level they agree better because the rounded 50% video level is 128.02 in PC levels so no dithering is needed. If you want to avoid this small error either use HCFR video levels end-to-end or use madTPG for PC levels.
Hi zoyd, thanks, that makes sense given the oscillatory nature of HCFR reading about the madVR calibration, which would be indicative of some rounding issue going +/- about the HCFR levels. However, what confuses me is that I took screenshots of madTPG pattern and HCFR pattern at 40IRE and looked at it in PS, and the Histogram mode indicated both having a mean of 102.0...which was unexpected. I was assuming that I would see slightly different levels due to dithering and how the video levels were being scaled to PC levels but I could not see this. Not sure why.

Quote:
Originally Posted by zoyd View Post
To test the truncate switch you should compare AVSHD to another disk that uses round nearest, again with video levels end-to-end to avoid the 0-255 mapped rounding errors.
I'm a bit confused by this... shouldn't the point of the AVSHD setting in HCFR be that the backend calculations should be adjusted so that this grayscale run should match my HCFR and madTPG runs? Otherwise, if I had calibrated using AVSHD in HCFR and called it a day, my actual calibration would be quite off when viewing videos using madVR.

Is this a problem primarily due to PC level calibrations? If I had done all my calibration with video levels, would AVSHD results at least matched HCFR results in that case? If not, I seem to be missing something as I would think that we would want to end up with the same calibration settings on the TV irrespective of which pattern generator we use (as long as the software is accounting for what software is being used for pattern generation).
orion2001 is offline  
post #4138 of 4431 Old 08-13-2014, 08:38 PM
AVS Special Member
 
HDTVChallenged's Avatar
 
Join Date: Jan 2002
Posts: 8,404
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 134
Quote:
Originally Posted by orion2001 View Post
I'm a bit confused by this... shouldn't the point of the AVSHD setting in HCFR be that the backend calculations should be adjusted so that this grayscale run should match my HCFR and madTPG runs? Otherwise, if I had calibrated using AVSHD in HCFR and called it a day, my actual calibration would be quite off when viewing videos using madVR.
No ... the point of "the setting" in HCFR is simply to match HCFR's expected luminance targets to the luminance targets encoded on the various test disks. Nothing more, nothing less.

PS: Keep in mind that HCFR is very much a "work in progress" ... by an army of one programmer (other than the Argyll contributions) ... for free ... It's unreasonable to expect Zoyd to anticipate every "unique" program usage scenario ... some assembly may be required.
HDTVChallenged is offline  
post #4139 of 4431 Old 08-13-2014, 11:06 PM
AVS Special Member
 
madshi's Avatar
 
Join Date: May 2005
Posts: 5,471
Mentioned: 6 Post(s)
Tagged: 0 Thread(s)
Quoted: 61 Post(s)
Liked: 129
Quote:
Originally Posted by orion2001 View Post
However, what confuses me is that I took screenshots of madTPG pattern and HCFR pattern at 40IRE and looked at it in PS, and the Histogram mode indicated both having a mean of 102.0...which was unexpected.
Good point. This might indicate that HCFR is rounding the desired test pattern colors only *after* conversion to PC levels and not before? Or maybe before *and* after conversion to PC levels? Both would explain why you measure madTPG to show exactly 102.0. If HCFR rounded only *before* conversion to PC levels, but not after, then madTPG should display 102.466 which (according to your tests) it doesn't do.
madshi is online now  
post #4140 of 4431 Old 08-14-2014, 06:45 AM
Senior Member
 
orion2001's Avatar
 
Join Date: Jun 2008
Posts: 470
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 272 Post(s)
Liked: 159
Quote:
Originally Posted by HDTVChallenged View Post
No ... the point of "the setting" in HCFR is simply to match HCFR's expected luminance targets to the luminance targets encoded on the various test disks. Nothing more, nothing less.
This, I don't understand. What is the point of keeping track of expected luminance targets based on differential encoding of patterns if the end result is not to obtain similar calibrations using any of the supported test pattern sources? Let's say we know that 40% gray on AVS709 actually translates to 39% (for simplicity sake) in PC Levels due to the way it is encoded. If HCFR supports AVS709 and is aware of the way it is encoded (which it does), then shouldn't it internally determine the luminance target assuming this is a 39% gray pattern, not a 40% Gray pattern? That way, you have a much better chance of tracking correctly when switching between AVS709 and the HCFR or madTPG pattern sources. My guess is that perhaps it does this correctly for video levels and that using AVS709 v/s HCFR pattern for video levels would result in the same curve, but I have no way of testing it myself.
Quote:
Originally Posted by HDTVChallenged View Post
PS: Keep in mind that HCFR is very much a "work in progress" ... by an army of one programmer (other than the Argyll contributions) ... for free ... It's unreasonable to expect Zoyd to anticipate every "unique" program usage scenario ... some assembly may be required.
Oh, don't get me wrong. My persistence in this matter is not a case of me being ungrateful or feeling entitled to anything. I really appreciate zoyd's efforts and would happily donate if he were willing to accept donations for his efforts. I just want to understand a) if I am using the software incorrectly or if there is a way to have my calibrations using AVS709 match up with HCFR/madTPG and b) If there is some bug or something missing, perhaps it might be a help to zoyd in that it could be something he might look at/implement if he felt it necessary.

Quote:
Originally Posted by madshi View Post
Good point. This might indicate that HCFR is rounding the desired test pattern colors only *after* conversion to PC levels and not before? Or maybe before *and* after conversion to PC levels? Both would explain why you measure madTPG to show exactly 102.0. If HCFR rounded only *before* conversion to PC levels, but not after, then madTPG should display 102.466 which (according to your tests) it doesn't do.
What zoyd posted about HCFR rounding down after conversion to PC levels and madTPG not doing this, would make a lot of sense in interpreting my results comparing HCFR v/s madTPG. Unfortunately, that doesn't square away with the fact that my screenshots indicate both outputting 102 level for 40% Gray. This also doesn't then explain why I am consistently measuring a gamma difference as seen in the plot. What I can think of is:

1) This is user error on my part in taking screenshots? Perhaps madTPG is in fact showing 102.466 via dithering while HCFR is showing 102, thus accounting for the discrepancy as per zoyd's explanation. Not sure why my screenshot wouldn't show this though.

2) HCFR calculations assume that madTPG is showing the non-rounded PC level via dithering, i.e. 102.466 when in fact it is showing the same rounded level as HCFR - 102 and is compensating for that in the calculations resulting in the difference. However, I don't believe this is the case either as I recall the actual meter luminance reading being different for the 40% Gray HCFR v/s MadTPG measurement. I'll go back and check my measurements in the evening to confirm.

3) Even though window sizes are set to be the same, slight differences may impact ABL on the plasma differently resulting in this slight mismatch between the two. I don't think this is the case though, as I have measured 8% v/s 12% windows using madTPG and in both cases the measurements line up pretty much identically.

I don't mean to be a pain in the ass, going on about this. I'm just curious to understand the source of what I am observing and in providing any useful feedback that may be of help to zoyd in the future. Cheers
RyanHomsey likes this.
orion2001 is offline  
Reply Display Calibration

User Tag List

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off