AVS Forum banner

The cheap Test Pattern Generators accuracy thread

198K views 1K replies 134 participants last post by  Brho0m13 
#1 · (Edited)
Finally I could satisfy my (and I hope your) curiosity to know which test pattern generator (or TPG) in the low price range is really accurate. To do this I bought a medium-high end TPG: DVDO AvLab TPG, which has inside a signal analyzer whose accuracy has been verified and confirmed with an Accupel DGA-6000 by the legendary Maciej Koper.

Let's start! Here are the contenders:



Raspeberry Pi 4B with PGenerator by Riccardo Biasiotto and Juan Loya with the indispensable help of the PGen team


Apple TV 3rd Gen (Model A1427)


Apple Media Center of two generations ago. Can be used with MobileForge (CalMAN) and LightSpace Connect (LightSpace). The limit of this object is the adjustable refresh rate, which is limited to 50 / 60hz, no 24hz.

Apple TV 4K


The latest Apple release. Can also be used with MobileForge (CalMAN) and LightSpace Connect (LightSpace).
On the spec sheet a bomb...

NOV 2022 UPDATE: as of today, no version of the Apple TV 4K is bit perfect because some kind of dithering is applied on its output. That includes the 2022 version. Use RGB high output setting.

Amazon FireTV Stick


Very economic and portable object, with a more than decent video quality. Can also be used with MobileForge (CalMAN) and LightSpace Connect (LightSpace). Here the limit is that the output is only one: RGB limited (16-235), no YCrCb

Raspberry Pi 3 B + with PGenerator by Riccardo Biasiotto


A tool that can really do everything! The only limit is the imagination!

Is my laptop bit accurate? Dell Inspiron 17-7779

...and my Blu-ray disc player?!? Panasonic DMP-BDT260


Before moving on to the test results and their commentary, I would like to define what the reference TPG should be... easy, it has a name: Accupel DGA-6000! It is perfectly accurate in every color space (supports up to Rec2020), with every color depth (8/10/12 bit), at every resolution (up to 4K) and frequency (24/25/30/50/60), both in SDR and HDR. It has one single flaw: it lacks of Dolby Vision support, but we know whose fault it is!
Limiting the test to SDR, I can state that a reference TPG must be bit accurate in YCbCr at 1080p resolution with all refresh rates, in this case the reference color space is Rec709. It is therefore in this "enclosure" that I have carried out my tests. Where a device resulted not accurate in YCbCr (or the YCbCr output setting is not present) I have tested its accuracy in RGB Full or Limited but with 16-235 videoscale. The display used is a TV Monitor Samsung SyncMaster P2370HD which, despite obsolescence, it has played its irrelevant task very well.

How I did the tests: DVDO AVLab TPG powered via USB from the laptop, TPG under analysis connected (one at a time) to the HDMI input of the DVDO with the HDMI output connected to the display, CalMAN 2018 Ethusiast R3 Beta Build 27, LightSpace CMS HTP 9.0.0.2448.

09.23.2022 UPDATE: As of today, the most accurate, complete and cheap TPG is the Raspberry Pi 4(B) with PGenerator software. It can output bit perfect 8/10* bit SDR/HDR and DoVi** patterns in RGB and YCbCr and it is FREE (how stupid is that :D), you just have to buy the hardware which isn't expensive at all.

* ColourSpace is the only calibration software offering 8/10/12/14/16 bit depth patterns. All the others offer 8 bit only.
** DoVi is only possible with ColourSpace software

There's also a Ted's version of the PGenerator which has these advantages (ask @ConnecTEDDD for further information):

1) 150+ Higher quality YCbCr 4:4:4 or RGB patterns, static or flashing, with the capability to change the flashing period. (compared to the YCbCr 4:2:0 patterns of the BD/Media Files).

2) Brightness Patterns without text.

3) Quick Black Patch Injection to override your current selected patch from the calibration software. (Ted's PGenerator Template)

4) Custom Library Upload capability. (upload custom 1080p patterns)

5) Ted's PGenerator users have access to 8 and 10-bit patchlists for ColourSpace/ColourSpaceZRO for grayscale/saturation/luminance/ColorChecker/quick profile (to check RGB Separation using ColourSpace ZRO)

6) For ColourSpace/ColourSpaceZRO, a free license enhancement to inject patterns using ColourSpace/ColourSpaceZRO GUI.

7) Conversion of Ted's SDR patterns (static/flashing bars) to HDR10/HLG or DV. The near black patterns are the only ones recommended for that conversion for the near black evaluation. It will work with custom uploaded patterns also.

If you were out to know what to buy, you just found out! The rest of this post is about other less complete and/or expensive solutions or a nice walk through history! Enjoy.

TPG for CalMAN:

Ted's Disc Media Files with third generation Apple TV
YCbCr 444 8 bit


MobileForge with Apple TV third generation
YCbCr 444 8 bit


RGB High


...and RGB Low


MobileForge with Apple TV 4K
YCbCr 444 8 bit


MobileForge with Amazon FireTV Stick
RGB Limited 8 bit


RPi PGENERATOR by Riccardo Biasiotto
RGB Full 8 bit


11.21.2018 UPDATE: As requested by @ebr9999 I tested RGB limited output of RPi PGenerator. As previously stated, the not so bad results still prove that RPi is bit accurate only with RGB Full output with LightSpace or CalMAN videoscale set to 16-235.

Here it is:


06.24.2019 UPDATE: As requested by @ebr9999 I tested RPi PGenerator YCbCr limited output, videoscale set to 0-255. Again, as stated above, the results still prove that RPi is completely bit accurate only with RGB Full output with LightSpace or CalMAN videoscale set to 16-235. But, what is better for you, it depends on your TV RGB ---> YCbCr rounding errors.

Here it is:

END OF 06.24.2019 UPDATE

Dell Inspiron 17-7779 with CalMAN built-in TPG
RGB 8 bit


Ted's bluray disc with Panasonic DMP-BDT260
YcbCr 444 8 bit


Conclusions: As shown by the graphs, at the moment, MobileForge is not bit accurate with every device even if the output of the latter is (as in the case of Apple TV third generation, as proven by the test with the media files of Ted's disk). MAYBE wrong triplets are sent. With Apple TV 4K the situation seems to improve a bit but it's just an illusion. The Apple TV 4K, in fact, with output YCbCr and RGB seems to process the image continuously returning fluctuating values with digital errors ranging from a theoretical minimum of 1 to a theoretical maximum of 6 per triplet (as shown in the chart in which Ted patterns are used). For this reason, even if I reported the minimum values (by looking for them pixel by pixel) I did not calculate the relative dE for a simple reason: Apple TV 4K is not effectively usable as a TPG. That's why I will not proceed to test it further.

11.13.2018 UPDATE: Obviously, if ATV 4K is the only TPGable device you own you have no choice than use it. My only recommendation is to NOT measure the patches produced by it in contact mode, the more patch area you measure the more the digital errors will be reduced... forewarned is forearmed.
Since there are still some doubts about ATV 4K behavior with YCbCr/RGB high output, I filmed two videos, both displaying Ted's Color Checker Classic Media file via Infuse 5 Pro App with:
1) ATV 4K at 1080p 24 Hz YCbCr output
2) ATV 4K at 1080p 24 Hz RGB high output

09.19.2019 UPDATE: I've checked again AppleTV 4K YCbCr/RGB High with tvOS 13... it still keep the same behavior filmed here for YCbCr and here for RGB High

That confirms my belief that it's not a software issue.

02.28.2020 UPDATE:

it'll be a quick update! I did some testing on an Apple TV FullHD (4th Generation) and an Amazon Fire TV Stick 4K:

- Apple TV Full HD is bit accurate only with RGB output. YCC is not accurate with YCC content or if used as TPG.

- Amazon Fire TV Stick 4K has YCC output bit accurate but can't be used as TPG because I tried with LS Connect and I didn't get bit accurate results.

So, we can use Apple TV Full HD as TPG only with RGB output, while Amazon Fire TV Stick 4K can't be used as an accurate TPG at all.

05/22/2021 UPDATE: see HERE

NOTE: Except Rpi, I tested all devices with Ted's Color Checker Classic media files (or BD for my BD player) and all of them showed solid triplets except ATV 4K.

A very nice request to analyze CalMAN built-in with my laptop HDMI set to YCbCr has been made by baller02, I didn't think about it when I ran all the tests so I've analyzed Calman built-in TPG only with my laptop HDMI set to RGB and it turned out to be bit accurate. So prior to analyze CalMAN internal TPG in YCBCr, I have to do it with Ted's patterns and... Read this post for more.

The Raspberry Pi (RPi) PGENERATOR by Riccardo Biasiotto is bit accurate and very stable (in RGB Full, not in YCbCr). We're talking about a DIY solution made by a NON PROFESSIONAL. Really outstanding, especially because it is 360° configurable, it connects via Ethernet, Wi-Fi, Bluetooth and it works with every calibration software I know!

Surprisingly, the HDMI output of my laptop resulted bit accurate!!!

TPG for LightSpace

LightSpace Connect with Apple TV third generation
YCbCr 444 8 bit


LightSpace Connect with Amazon FireTV Stick
RGB Limited 8 bit


RPi PGENERATOR by Riccardo Biasiotto
RGB Full 8 bit


Conclusions: Apple TV 4K has been excluded for the reasons mentioned above. All TPG solutions for LightSpace are bit accurate. The choice here could fall on the third generation Apple TV which is also perfect in YCbCr. On the same level, the RPi PGENERATOR by Riccardo Biasiotto.

Final Conclusions and tips: For those who only use CalMAN or even LightSpace, the only affordable and accurate bit solution is the RPi PGENERATOR by Riccardo Biasiotto.

For those who only use LightSpace and own a third-generation Apple TV, they can download and purchase LightSpace Connect but it is strongly suggested to use it connected via an Ethernet cable. Wi-Fi, during the generation of a color cube, could disconnect making you lose happy hours of real life. The RPi is also highly recommended with a super stable wireless Bluetooth connection.

Please, let me know your opinions, suggestions and (why not?!) your own tests!
 
See less See more
22
#2 ·
Reserved
 
#3 ·
Reserved 2
 
#6 ·
Will do. Probably tomorrow morning. But I suspect I will get the same floating results.
 
  • Like
Reactions: MrRobotoPlus
#5 ·
Great work Miki thanks for your efforts thought you're were a lazy Italian like I am :)
 
#7 ·
:D
I am! You don't know how many test I could have done and how much I could have written but my laziness made me desist almost immediately! lol
 
#9 ·
Sure Steve, I will write you my address so you can send me those various devices to test. Oh and a Klein K-10A and a CR-300, thanks! :D Just kidding ;)

What other devices would you like me to test?
 
  • Like
Reactions: L30Z3N and jrref
#11 · (Edited)
It needs another device (HDFury Integral/Vertex).

What did emerge once again from my testing it is how Ted's disc is a MUST in a calibrator's "library". It was and still it is a necessary tool in manual (enthusiast) calibration but it is also needed when you don't know if one of your device is bit accurate. I'm thinking about laptops HDMI output for example.
 
#19 ·
If you added the HDFury Integral/Vertex to the Rpi would the HDR Test patterns be accurate or does that still need to be tested?
 
#12 · (Edited)
It would be interesting to put together a list of PC output solutions that work reliably...

Currently I use the Masciola HDR10 patterns which are very nice.

Anyone try using an Intel NUC7 to run CALMAN, and to generate 4K HDR patterns via HDMI 2.0 output?

ERic
 
#13 ·
Miki,

Please test the RPi PGENERATOR by Riccardo Biasiotti in YCC, it will help debug the current YCC output problem...

E
 
#29 ·
#14 ·
In any case, we really need to find a low-cost 4K HDR solution
 
#16 ·
Why? Aren't you happy with 1080p HDR TPG?
 
#15 · (Edited)
I tested it and I already spoke about it with Riccardo. It is technically easy to fix but it requires a lot of time because a analysis color by color is needed as a subsequent triplets compensation... we are talking about at least 9,261 triplets (21^3 cube points). It's a six months job: 50 colors per day during free time.
 
#17 · (Edited)
Worry that color would be off...
Need easy reliable solution that puts TV in correct mode. 4K HDR10
Own integral, but too many boxes too much setup for std calibration

E
 
#18 ·
I wouldn't worry so much about that. First because HDR10/DV calibration is all about grayscale as touching the CMS for HDR10 introduces artifacts while Color Gamut calibration for DV (eg. DV file tweak) gets color even more off, second because you and I don't have reference meters, so we have at least a dE of 2 already from the "Ready? Go!". This hobby is fun and educational but when it comes to have the job really done a good pro calibrator is the only really CHEAP solution.
 
#21 · (Edited)
Your assertion about DV config file is a misunderstanding of the differences of the 2017 vs 2018 LG panel. The panel saturation control that was introduced in the 2018 panel results in much better DV calibration results as far as gamut accuracy. One workaround on the 2017 model, Is to tweak the global saturation control after loading the config file. Most likely you will have to boost the saturation a little bit to get the color checker to fall in line.

Tyler
 
#20 ·
It would need to be tested. I might do that for 1080p SDR but then we should introduce a new dogma in our lives "with HDFury Integral what is bit accurate for SDR , it will also be for HDR" and trust it until someone with a 2k/4k HDR signal analyzer will confirm or destroy our dogma. I admit my ignorance here, I don't know if DVDO is able to analyze a HDR signal, I guess it can not, but @ConnecTEDDD surely might clear this thing up.
 
  • Like
Reactions: jrref
#25 ·
Thanks mate. It’s all about that: knowledge and brotherhood.
 
  • Like
Reactions: L30Z3N
#24 ·
@connectTEDDD

You pointed out something very interesting that is often overlooked:

"The problem is that ideally you need to generate YCbCr REC.2020 patterns when you perform HDR10 calibration (and YCbCr REC.709 when you perform SDR calibration) because this is the colorspace where your player will output (bit-perfect) when you will playback your movies.

TV's can have different results when you send to a their HDMI Input YCbCr or RGB signal (colorspace conversions will be involved/processing...you can have different near black or RGB balance etc.), the ideal is to send always the same colorspace as your source will output later during movie playback, here is the advantage or an external generator, the guarantee that you can have accurate patch generation."


It would be interesting to test some of the more popular high end sets from LG and Sony, and Panasonic if you are not in the US, to see what the difference is, if any to the calibration sending the set YCbCr and RGB. Since the TV won't process RGB, there should be no error but converting YCbCr to RGB, maybe some sets have a problem?

This is a pet interest of mine because through experimentation on the LG and Sony OLEDS, I can never see the difference putting the Murideo to Auto vs YCbCr or RGB. At least nothing that I can detect. Everyone says don't use Auto because the EDID could be wrong from the TV, IDK. I think a lot of us assume high-end TVs do the conversion from YCbCr to RGB correctly so there shouldn't be a difference.

What would be the best quick and long test to see where there is a potential for errors n this regard?
 
#26 ·
It would be interesting to test some of the more popular high end sets from LG and Sony, and Panasonic if you are not in the US, to see what the difference is, if any to the calibration sending the set YCbCr and RGB. Since the TV won't process RGB, there should be no error but converting YCbCr to RGB, maybe some sets have a problem?
Why do you think a TV won't process an RGB input? As far as I know this is TV dependent.

It is my understanding that very often the image processing in a TV actually operates in YCrCb colourspace; so RGB input could result in more conversion than YCrCb input.

YCrCb video from content -> player converts it to RGB -> TV converts RGB to YCrCb -> TV processing -> TV converts it to RGB for the panel....

Vs

YCrCb video from content -> TV processing -> TV converts it to RGB for the panel....

(there is some non-specific coverage of the issues here: http://spearsandmunsil.com/portfolio-item/choosing-a-color-space-2/)
 
#38 ·
Exactly. But will the TV convert anyway your RGB only video source to YCbCr and then again in RGB?
 
#40 ·
The point is to have always the whole video chain calibrated, so to include all colorspace conversions to the signal that will appear to the panel....to be able to measure it with the instrument.

When movies are mastered, they calibrate the whole video signal chain also, to include to the correction everything, video card output....software generate patches from inside color grading window etc.
 
#39 ·
So, now I'm more confused than ever.
For ATV 4K what is better?

- Set the ATV to YCbCr then make a calibration within infuse with Ted's patterns for SDR and Masciola for HDR ?
- Make a calibration in RGB with the Raspy > Set ATV to RGB and then maybe checking the errors with patterns within Infuse?

:confused:
 
#41 ·
The best and quickest test is to measure video streamed patterns from inside ATV and then measure the same patterns via Rpi. (Your TV don't need to be calibrated, just use the default settings)

Compare then both reports and decide.
 
#49 ·
It’s useless to measure. Color values (YCbCr/RGB triplets) fluctuate always, in menu, in movies, in infuse, in airplay. ATV 4K it’s not usable as TPG. Do. Not. Use. It.
Is it clear now? 😄
 
#53 ·
Thank you for the basics. I know how to perform a calibration, at least. And I do that at any firmware update.
Actually that wasn't my question.
 
#54 ·
I’m sorry, would you be so kind to reformulate your question for me please?
 
#55 · (Edited)
Since WE already know all the problems about ATV output. What is still better to calibrate taking in consideration the space color conversion process chain, ycbcr or rgb?
 
#56 ·
As the values aren't there to look at, can I ask how bad is the RPI YCbCr for calibration as there isn't a graph for it? Are we talking just minor errors, or great big monster errors that are very obvious?

And using the Apple TV, I assume that you mean using the Lightspace Connect app on the iphone/ipad and then screen mirroring to the AppleTV? I couldn't find a specific Lightspace Connect app in the AppleTV app store.
 
#57 ·
RPi outputting YCbCr returns monster errors, 14/35 digital errors I have the chart but I didn’t post it because it could have confused people easily.
LightSpace Connect doesn’t have a native ATV app. So, yes you must use AirPlay with it. But again,I would use it with ATV 3rd and maybe 4th gen, not 4K.
 
  • Like
Reactions: vollans
#61 ·
 
Top