Originally Posted by Erik Garci
The quote above is from the earlier thread. It seems that the 10-bit LUT is the bottleneck that limits the VideoEq's output to just 10 bits (padding the lowest 2 bits with zeros).
Now that the VideoEq is being released, maybe someone can post a more detailed answer to my question.
The VideoEQ uses a 10-bit LUT with values from 0-1023 in the format:
0000 = ,0000,0000,0000,0000
0001 = ,0001,0001,0001,0000
0002 = ,0002,0002,0002,0000
0003 = ,0003,0003,0003,0000
0004 = ,0004,0004,0004,0000
0005 = ,0005,0005,0005,0000
0006 = ,0006,0006,0006,0000
0007 = ,0007,0007,0007,0000
0008 = ,0008,0008,0008,0000
0009 = ,0009,0009,0009,0000
0010 = ,0010,0010,0010,0000
I believe the CMS matrix is calculated in 12-bit and is stored as:
R000 = ,004096,000000,000000,000000
R004 = ,000000,004096,000000,000000
R008 = ,000000,000000,004096,000000
R012 = ,000000,000000,000000,000001
G000 = ,000000,000000,000000,000000
G004 = ,000000,004096,000000,000000
G008 = ,000000,000000,004096,000000
G012 = ,000000,000000,000000,000001
B000 = ,000000,000000,000000,000000
B004 = ,000000,004096,000000,000000
B008 = ,000000,000000,004096,000000
B012 = ,000000,000000,000000,000001
C000 = ,000000,000000,000000,000000
C004 = ,000000,004096,000000,000000
C008 = ,000000,000000,004096,000000
C012 = ,000000,000000,000000,000001
M000 = ,000000,000000,000000,000000
M004 = ,000000,004096,000000,000000
M008 = ,000000,000000,004096,000000
M012 = ,000000,000000,000000,000001
Y000 = ,000000,000000,000000,000000
Y004 = ,000000,004096,000000,000000
Y008 = ,000000,000000,004096,000000
Y012 = ,000000,000000,000000,000001
I'm not very familiar with CMS matrices and there are no tools available from AV Foundry yet to adjust the CMS, so I don't know how it actually works, and haven't had time to experiment with it.
I imagine that it is processed in 12-bit and then rounded down to 8/10-bit for output.
The output format is limited by the input format, so if you have 10-bit going into the VideoEQ, you get 10-bit out. If you have 8-bit going in, you get 8-bit out. (even though the LUT is 10-bit) I can only use RGB in my setup, so I don't know if YCbCr will count as a 12-bit input.
Currently, I've just spent a fair bit of time this afternoon taking measurements and adjusting a LUT by hand from 0-15% in 1% steps, and then 15-100% in 5% steps to ensure that if there is posterisation visible, it is not a result of how they were being calculated below 10%.
I have the data, but I need to create another spreadsheet that will turn the 33 points of data I have into a 1024-point LUT. Once I've done that, I'll be able to determine whether or not I keep the VideoEQ in my video chain.
Having the display this linear is amazingmy CRT has never looked this good. However, the amount of posterisation I've seen with my previous LUTs is too much to ignore. If this doesn't reduce it to an acceptable level, I'm not sure I want to keep it. One of the main reasons I'm still using CRT now is that any digital' display I've had/seen has shown too much posterisation.