Setting contrast with Spears and Munsil disc - Page 15

I suspect the effect will be much more subtle . The pattern has zero fall off on the edge of the checker so its going to ring like a bell , pardon the pun. You never get edges like this in real photography: even the sharpest edges have a falloff of about 5 pixels and thats not even accounting fro aberration.
It does show that there is definitely a likely mechanism for generating rather large excursion from rigorous 16-235 material just from the resizing steps ( from both the resolution change and the chroma down/upsample requirements).




The pattern is generated directly with the upper limit set at 235 including the colorwheel( which is actually hitting 236..this might be a rounding error as its processing in 16bit integer and I'm just eyeballing the 235 with a pixel analyser rather than getting it accurate to 4 decimal places) so there is no clamping stage afterwards to make it toe the line.

When I use the real image I'll do two versions. One that targets 235 as the rigid white point and one that targets 254 but with roll off from 235 so that if its clipped it still looks acceptable. It will probably be the Kodak LAD chart.

triangle filter 4:2:0 ( the forum is converting the pngs to jpeg for some reason...maybe we've exceeded our storage for the thread , the images do illustrate what the raw imagery is doing though...I can upload the pngs to John if required)

The triangle filter produces little if any appreciable ringing. If I use it just for the rescale to 1080p I get no extra level generated above 235* or below 16.

*allowing for the fact that the original colorwheel does actually hit 236 as previously stated.

The 4:2:0 downsample and upsample does produce significant level above 235 ( all the way to 255) however only where color information exists ( naturally).



same image 235 and below clamped to black (note level generated "entirely" on color information , white checks are clear )

Thanks for doing the tests. Will be interesting to see if it goes all the way to 255 on the Kodak chart.

John

real imagery this time.
The Kodak LAD chart (2k , 1828x1332 10bit log) (I'd much rather have a fullap 2k scan than this academy one so I'm going to do some hunting about beore continuing but this is an example of log to video conversion)

This is a crop from the original 10bit log albeit downsampled to 8bit precison so you guys can see it. This is a 35mm negative full density scan. Note the huge detail in the whites.



This is the same image converted to video . I graded this by eye with a subjectively created curve referencing a rec.709 display. (its just a single curve for RGB I have not rebalanced RGB or messed with HS directly)

This is somewhat like the process film goes through to be converted to video. I have not clipped it at 16-235 but let it roll off normally according to how I've visually graded it.

I have transferred ALL the detail from the negative (I have not clipped it at 10bit logs 685 white ref ... this tallies with what I have seen comparing modern film masters with the original 10bit log scans). The notion of clipping at white ref has gone out the window and the colorist is at liberty to use all the available range.

This is the process I normally use to convert log to pleasing video imagery. At no point did I bother to meter the level above 235 I just rely on the rec.709 environment to drop the white levels in appropriately in conjunction with what I visually decide and the fact that I'm somewhat trained visually to not bang off the endstops too much.

Thank you, appreciated!

One thing to talk about would be the scaling algorithms used. Are studios really using sinc for downscaling chroma? Are DVD/Blu-Ray players using Lanczos to upscale chroma? Which algorithms do studios use to downsample luma? My personal opinion about these are:

(1) Are studios not likely to choose a soft filter for chroma downsampling? I wouldn't expect them to use sinc here.

(2) I think DVD/Blu-Ray players are also likely to choose a soft chroma upsampling algorithm, maybe even the very soft Gaussian. I know that ATI and NVidia use very soft upsampling filters for chroma. Haven't really checked DVD/Blu-Ray players, though.

(3) For luma my impression is that different studios are choosing very different downsampling algorithms. E.g. compare these two screenshots from the Minority Report Blu-Rays:

Fox (EU): http://pic.phyrefile.com/s/st/st0ni/2010/04/08/1.png
Paramount (USA): http://pic.phyrefile.com/s/st/st0ni/2010/04/08/2.png

Notice how there is clear ringing present in the Paramount image, while there is none at all in the Fox image. The Fox image also has a finer grain reproduction. However, the Paramount image has smoother edges. The Fox edges seem to be a bit "jumpy". Also check out the name plate on Tom Cruise's jacket. It's supposed to read "Anderton", IIRC, and it looks quite different between Fox/Paramount. My personal opinion is that Paramount used Lanczos or Sinc to downscale the image, while Fox probably used simple decimation (simply dropping every other pixel row/column).

I think if with our experiments we want to try reproducing the WTW pixels of existing consumer material, we need to try using the same algorithms, studios are likely using. Which means that using Sinc/Lanczos for chroma sub/upsampling is IMHO not a good choice. I'm torn on which Luma downscaling algorithm to choose, though, because studios don't seem to agree on the best algorithm for that, either...

Your opinions?

@JohnAd, not sure if it makes sense, but I just thought, maybe you could extend your test software a bit. You could sort WTW pixels into 3 different categories:

- invalid source material (if the original Y channel is already > 235)
- probably bad WTW pixels (if they're near a luma edge)
- possibly real WTW detail (all remaining pixels)

Not sure if it's worth it to do that, though. For edge detection you could check the local covariance or something like that...

Do you yet have a view how the current blu-ray process must work in order to avoid the >235 pixels in the demo grade you just did? On that one a lot of hair detail goes if clipped to 235.

I've just been playing around with gimp and I don't know if this will help illustrate what I mean, this is a very quick and dirty attempted blu-ray style grade of the same image, rather than clip I've compressed the top end whites quite a lot but there is still some detail and it does top out at 235. So the transfer function is an exagerated S shape. As it's starting from an 8bit log there is a lot of gaps in the histogram


Do you think this type of white compression is what happens in practice? Or something else?

John

This is the crux of the matter. When I grade I don't explicitly aim for 235 as the white point. My rec.709 environment shows detail all the way to 254. Most colorists I've seen if grading subjectively will not be trying to hit a number, they will grade the material to look subjectively pleasing within the definitions of their reference environment.

IF the reference environment pegs out at 235 their imagery will top out at this point.

What I want to figure out is if they actually work to a rigorous 235 and the level we additionally see is processing noise or if its an actual extension of real detail that was deliberately placed by the grade.

What I intend to do is create a version that isn't locked to 235 ( as above) and the run it through 1080p and 4:2:0 processing and do the same with a version that is topped out at 235 and see what one seems similar to the BD examples in terms of level above 235. (which to my reckoning looks on the whole like noise).

To create an image with no level above 235 I would impliment a secondary correction that maps peak white to 235 as far as the reference environment is concerned and then subjectively grade with that switched on. I won't grade for 254 and then do a quick tweak to compress it down to 235 (although with 32 bit precision and a 10bit log source its some what moot how I get there).

If you look at the imagery from BD its probable that the image I created if clipped at 235 wouldn't look that disimilar to the highlight behaviour we are seeing anyway.( regard video as very clipped looking anyway in highlight behaviour so I may be a little bit more preseving of white detail than most colorsists are).

So at the moment i can't tell whether they target 235 explicitly or let it run up to 254 and trust to their mastering aesthetic to not place too much important detail up above 235 so that if/when its clipped its of minimal concern. I suspect most colorists are not necessarily aware of their white point in terms of absolute system setup so ironically they are not always the best people to ask.

I generally find the sinc is VERY popular for downscaling generally even further up the chain from the video mastering. There has been a bit of an arms race in recent years as to who can produce the sharpest looking imagery and its generally led to over aggressive use of sharpening techniques.

I'll have a quick look around and see if I can find out any more but I'm pretty certain sinc is widely used.

You are probably correct that its not used for chroma downsampling.


Another thing I've noticed is that some modern masters seem to be direct 1920x1080 crops from the 2048x1152/1556 2k DI so what you get is 1:1 pixel with the 2k but a small crop in on the original aspect ratio. Discussed in a thread on here a while back. obviously this doesn't hold true for scope.

Yeah, sinc downsampling is sharp. But I'm not sure I like the look of grain sinc produces.


Minority Report was reportedly remastered at 4K. Have you looked at the screenshots I posted? Which one do you prefer, especially in terms of grain reproduction? The Fox or Paramount one? I always thought sinc/lanczos would be better for downscaling compared to simple decimation. But after seeing those Minority Report screenshots, I'm not sure if decimation isn't a better idea, at least for grainy movies. But that's a bit out of topic, I guess...

I've not had time to look but Minority Report might be a little bit of an extreme example. I recall they added post grain over the previous video master to better retain the desired hefty grain look of the film version.

I'm not a fan of overly sharp filters. I tend to only use the sinc filter for downscaling and even then I often switch to a lanczos which whilst softer still tends to behave better even with a bit of simple unsharp mask afterwards.

So, are you guys getting closer to a generic recommendation of where to set Contrast via the S&M white bars pattern? Show all the bars up to ~250+, or set Cst higher so that only those up to 235 (plus maybe a few) are discernable?

The complete lack of genuine WTW found on those discs tested gets me to thinking that there might be some kind of automated "legalising" process that gets run somewhere, now that may be simply a clip but I'd have thought that would look more obvious.

John

+1, I would appreciate a summary of what has been determined up to this point, preferably in layman's terms as some of the stuff discussed in the latter pages of this thread is above my head.

See I'm not so sure it would look much more obvious. Video to me always looks "clipped" in the whites. I've not seen a grading suite that hard clips at 235 yet however thats not to say that the actual data isn't clipped further back from the display : for example if its linearised with a hard 235 white point.

That imagery I sent you a while back ( that you can't post) that compared the log with the BD seemed to have everything up into white transferred to the BD. So if that isn't hitting above 235 with real detail then the master was definitely aiming for 235.

I do think from the data we've seen that its highly likely that there is nothing above 235 but "junk". I'm just not quite 100% convinced that it might not be possible something else is going on...yet.

Q:Is there level above 235 on commercial video?
A: Yes there is. In fact we have not come across one example where there was not level above 235 usually all the way to 254/255 at least somewhere.

Q: Is the level above 235 real picture information?
A: Not entirely answered this yet. Processing side effects can be responsible for generating level way above 235 even if the original imagery has no level above 235 prior to processing.

Q: Are there any technical reasons for displaying level above 235 if its only noise?
A: I say no , some others may disagree and may have a point.

Q: Does BD look better with 235 mapped to peak white on display.
A: I say yes.

Q: Does broadcast look better with 235 mapped to peak white on display.
A: Most of the time yes but sometimes it looks a little clipped to me: this may be the nature of the footage ie: it is just clipped.

Q: I'm getting fed up with all this esoteric navel gazing you guys are doing where the heck should I set my contrast control?
A: Try both 235 and 254 as peak white and let us know what you think.

As a quick round up here is where I think we are for hi-def discs

Data Examined:

Logs from ffmpeg with wtwfinder option from Blu-rays & HD-DVD discs, I've done a few, Mr.D has run a few too.

I've run:
Mamma Mia
American Gangster
Planet Earth Ep1
Syriana
HD Scape sampler

Mr.D has run
Transformers 2
Avatar
Pandorum
Fantastic Mr. Fox
The New World


Results:

All content tested has some pixels over 235 in both YUV and RGB

Examination of the shots with most WTW pixels from each source has failed to find any WTW pixels that cannot be explained by resize or chroma downsampling artefacts or burnt in subtitles

The exception is that the HD scape sampler does contain material with WTW content and clipping would be noticeable on this content.

Conclusion:

At the moment there appears to be little visible content above 235 on material not sourced from HD cameras.

Did some quick tests today, did find some visible WTW in channel 4's racing coverage and a tiny bit in some of the OB footage in today's endless election broadcasts, the studio stuff seemed to be better, supposedly that was all orginally HD.

John

Yes I suspect this is the case: I never see it on anything like high budget drama its usually news and sports footage and the occasional live studio report. (Q.I. sometimes looks a little blown out on Stephen Fry's shirts but it may well just be blown out with nothing over 235)

Its probably not worth keeping level over 235 for the occasional example.

I have been running with a 235 setup now for a couple of weeks and to be honest I do prefer the contrast boost . Every time I go looking for bit of clipped white info I'm sure I saw on BD I find there was in fact nothing there over 235.

Also try the contrast boost setting on a bit less than max, that might be a good compromise too, that will get you most of the boost and a little room at the top to avoid worst aspects of clipping when there is some WTW.

John

Thanks Mr.D and JohnAd for the summaries, as well as all your efforts. It has been most appreciated, certainly by me, and I'm sure many others.

Thanks for the summaries. It's good to know clipping at 235 doesn't negatively affect highlight detail in any major way. So, I guess now the question left is whether the data above 235 has any real value at all, even if it's just minimal.

So what does this say to ChrisWiggles and S&M, et al, who recommend quite strongly to show WTW all the way up?

So, final conclusion is to set top white at 235-240 instead of 254? Does it apply for all content sources?

I think as John says try a bit higher than 235 and you should get the best of both worlds. 240 might be a good place.

Display behaviour may also be a factor. If you can't set the contrast control high enough to clip lower down without trashing gamma or inducing massive runout you may have to reconsider your options.

Another option would be to run the hardware contrast as high as it will go without run out and use an off board CMS or similar lutting arrangement to map 235/240 on the incoming signal to the peak white on display and sort out any bumps in gamma grayscale or gammut. This tends to be how professional CMS profiling systems work.

Only interested in what the data is telling us , not starting fights.

Yes for video.

My PS3 maps peak white to 235 when super-white is off or RGB limited range is used (the RGB output is forced for XMB and Games by design of the PS3). My Xbox no longer does this when set to standard levels and YCC709 or RGB color space (through HDMI), but I don't seem to notice a color shift outside of test patterns, so it must not really matter.

Ok, thanks for the info and for all the time and work you've needed to reach this conclusion.

Although now I wonder why S&M, Chris Wiggles, and a good number of well known calibrators are so sure that WTW should be preserved all the way up to 254... are all they wrong?

My conclusion, at the moment, is that the pixels > 235 do have a value, clipping the >235 stuff does change the image. The issue is if the change, which is very small in practice, is enough to justify the loss in contrast. Different conditions will effect this trade off, basically the higher your black level and the more light in the room the closer I would push towards clipping at 235. If you had a pitch black room and display black at zero light then you should go all the way to 255. Also if your display pushes out enough light to get you anywhere near eye-fatigue territory then you should allow the WTW to be visible.

How this trade off applies I think is an individual decision and I think Mr.D has it right by suggesting that anybody who uses 255 at the moment try out 235-240 and anybody who currently clips at 235 trys out 255 (all that assuming your display doesn't do anything silly if you try it out).

John

Absolutely. Just try it and see and post back what your opinion is of running your setup either way.

Think of 235 as the "minimum" white ref . If you feel your display is a bit lacking in contrast you should feel at liberty to set the contrast control closer to 235 to improve things without worrying you are losing meaningful white detail by doing so.