Video pre-processing

[sspears]

I am starting a new thread to discuss pre-processing of video. In this context, I am talking about converting from a higher bitdepth down to a lower bitdepth. In the case of BD and HD DVD, it is often 10-bit 4:2:2 to 8-bit 4:2:0.

Below are two examples taken from the Elephants Dream project. You can download the publically available 8-bit uncompressed PNG files from their website. http://media.xiph.org/ED/1080-png/ Sorry, I don’t recall the exact frame number. I am not sure the process they used to create their 8-bit PNGs. Elephants Dream does have some pretty severe banding throughout.

I was able to obtain the original 16-bit OpenEXR files from them. In my two examples, the same path was used.
1. Open EXR file in After Effects CS3
2. Disable all processing
3. Export as 16-bit TIFF
4. Use Microsoft xScaler to convert from 16-bit TIFF (48-bit RGB) into 8-bit RGB (24-bit RGB)
5. The images were also converted from PC to Video levels.

In the left image I used one of the xScaler dither modes. In the other I used the xScaler rounding mode. There are a lot of different techniques to dither an image, this is simply our favorite. Don and I spent a lot of time working on this technology. It was originally written about five years ago.

The area to see the banding is on the light gradient to the right of the image.



It is simple to maintain this level of detail on a static image. The problem comes when there is motion. In this particular part of Elephants Dream, the image pans to the right. The dither does make compression more difficult in this example. I have other examples, like Coral Reef, where it actually becomes easier to compress.

[Kroenen]

That's very impressive Stacey.

You mentioned dithering can make compression more difficult when in motion. Paramount's HD DVD version of Beowulf, as you know has a lossy audio track while the UK versions have lossless.

Is it plausible that since the UK BD version most likely shares an encode with the UK HD version; that they decided to round down to 8 bit, instead of using xScaler, because the combination of the dithering and the allowance for the possible peaks of the lossless track, might have been beyond HD's peak bitrate limit.

[MovieSwede]

Kroenen I doubt the 1,5 mbs difference will have that much effect.

[sspears]

xScaler is SLOW. Its really only useful if you can run it in parallel. Microsoft never built an application that has incorporated it this way. One K and Deluxe have built their own parallel application that use it.

The example I provided is an extreme example. In a lot of cases, you can't see any banding from rounding.

You can increase the dither amplitude and reduce banding in the source, which is done by some.

[dr1394]

I've often thought that the best dithering would be some adaptive algorithm that only dithers on contours and leaves flat regions untouched.

Here's an image that's been dithered from 10-bits (by a very simple hardware algorithm) and also MPEG-2 encoded. The dithering helps the contours somewhat, but also adds noise to the flat regions.



Also, the resulting noise looks very much like DCT basic functions. I also wonder if there's a dither technique that takes into account how the dither will be compressed (by some DCT or DCT-like transform). Seems like a much more difficult issue though.

Ron

[Kroenen]

Ah gotcha. I was equating "compression being more difficult" with "requires a bitrate increase". I see what you're saying now.

Thanks Stacey. I appreciate the reply.

Thanks as well MovieSwede.

BTW Stacey do you have a blog?

[MovieSwede]

Kroenen

Thought I can add this.

While the difference between a lossy and lossless track is just about 1,5mbs in peak, there could be more stuff in the mix that effects peak and avarage bitrate.

But sspears can this stuff better then me.

[sspears]

I've often thought that the best dithering would be some adaptive algorithm that only dithers on contours and leaves flat regions untouched.

Ron, if you look at the pixel values in the dark part of the ED dither image, you will notice they are RGB 16, not a 15/16 or 16/17 mix of noise.

In the latest version of PEP, we included a contour correction filter. Its not perfect, but it's cool. I will ask Joe Kane if I can use an image from his new disc. It looks like Deluxe re-did the encode to fix some source side issues, so the banding is present. I had fixed it in the original encode I did.

[sspears]

Here is another clip. In the ED example, I converted from 16-bit RGB to 8-bit RGB. In this next example, I will convert from 10-bit 4:2:2 down to 2-bit RGB. OK, perhaps 2.5 bit. There are 5 levels used in the 2-bit examples. You can open in your favorite app and look at the RGB histogram if you don't believe it.

First up is the 10-bit to 8-bit reference image.

[sspears]

This is the 10-bit to 2-bit using rounding. If you look real close, you might be able to see some banding in the image.

[sspears]

This is the 10-bit to 2-bit using random noise dither. Its using an R250 random number generator. Its a bit noisy, but the banding has been reduced. I had to use a full quantization of random noise to reduce the banding.

[sspears]

This is 10-bit to 2-bit using error diffusion with the random number generator. This is using half a quantization of random noise to remove banding. This is our favorite and is what was used in the Elephants Dream example above.

[TheLion]

Impressive thread! Thank you for sharing, Stacey.

Can you provide any information if the recent BD release of Ratatouille (Deluxe?) used the xScaler dithering routine? Given the presence of really critical scenes (fog@ Saine...) I was pleasantly surprised by the complete (!?) lack of obvious banding artifacts.

[sspears]

Can you provide any information if the recent BD release of Ratatouille used the xScaler dithering routine?

Yes it did.

[TheLion]

Originally Posted by sspears

Yes it did.

Thank you. This settles it then - xScaler is very impressive indeed

[sspears]

So we have looked at dither, now I will post some image reszing samples using the same tool.

One of the ideas that we have been pushing is to perform all processing in linear light. The video you watch today has a gamma curve. Linear light involves removing the gamma curve, perform processing and then re-applying the gamma curve. When working with linear light, you need extra bits for processing. An 8-bit gamma corrected image would need 12-14 bits if working in linear light. Personally, I would not do it in less that 16-bit floating point.

The first sample is a zone plate that cycles from 5 MHz to 20 MHz. This is the original image, which is 1080p.

[sspears]

This image is the 1080p zone plate resized to 720p using linear light. The filter kernel is a bicubic algorithm.

[sspears]

This image is the 1080p zone plate resized to 720p like everyone else does it, ie with gamma correction. The same bicubic algorithm was used.



There are a couple of things to point out. First, the edges are darker. This is the highest frequency portion of the zone plate. Second, the moire is more pronounced. If you look closely at the linear version, the moire is present, just more difficult to see because the high frequency detail is not pulled towards black.

The linear zone plate looks more like the orignal. Again, same tool was used to resize, I simply turned linear processing on and off.

[sspears]

This next sample shows what happens on think diagonal lines as well as on strong colors. Pay attention to the thickness of the text and diagonal lines. Also look at the edges of the text on the red and blue backgrounds in the resized samples.

[sspears]

The image on the left was resized in linear light. The image on the right was done with gamma correction. Note that depending on your browser, and resolution, it might be top and bottom image.



You will notice that the "Here is some white text" has become thin on the right image. You should also notice that the green text on the red background has dark spots around the edges. You can also look at the thickness of the diagonal lines.

[sspears]

If you moved to a better filter kernel, something that is edge adaptive, you should get even better results. I have been pushing Silicon Optix and Anchor Bay Technology for a few years now to switch to linear light. The problem for them is cost.

As it stands, there is still a lot of room in consumer, and professional, products when it comes to image resizing.

[TheLion]

Impressive.

Stacey,

is a Lanczos-based resizing algorithm also supported with this tool or is bicubic preferred.

How much more processing power does linear light require compared to standard gamma corrected resizing? Is it "just" the difference between eg. 8bit and 14bit processing?

[sspears]

We have two filter kernels to choose from, Lanczos and Catmull-Rom. Catmull-Rom is our favorite. While it is an interpolation filter, by design, we stretch it to make it work as a low pass filter. With Lanczos we allow you to specify the number of lobes. We found that more than two lobes introduces ringing (a ring per addional lobe) on sharp edges, so we prefer two lobes, which is virtually identical to Catmull-Rom. When you see Lanczos2 and Lanczos3, it usually means number of lobes. I know several people seem to like Lanczos3.

How much more processing power does linear light require compared to standard gamma corrected resizing?

A lot more processing! The big performance hit is the conversion too and from linear light because it is a power function. I have attached an xls to convert between video levels, percent (gamma corrected) and linear percent. If you look at any of the linear cell's, you will see the linearize function.

[sspears]

AVIA, and other test discs, contain patterns that allow you to visually determine the gamma of the display. The patterns usually consist of single pixel black and white lines with a specific color gray background. These patterns only work if there is no image resizing being done, unless you are resizing in linear light!

[TheLion]

Originally Posted by sspears

We have two filter kernels to choose from, Lanczos and Catmull-Rom. Catmull-Rom is our favorite. While it is an interpolation filter, by design, we stretch it to make it work as a low pass filter. With Lanczos we allow you to specify the number of lobes. We found that more than two lobes introduces ringing (a ring per addional lobe) on sharp edges, so we prefer two lobes, which is virtually identical to Catmull-Rom. When you see Lanczos2 and Lanczos3, it usually means number of lobes. I know several people seem to like Lanczos3.


A lot more processing! The big performance hit is the conversion too and from linear light because it is a power function. I have attached an xls to convert between video levels, percent (gamma corrected) and linear percent. If you look at any of the linear cell's, you will see the linearize function.

Thank you, interesting indeed.

Personally I have always favored Lanczos3 - although I agree slight ringing becomes visible - with Lanczos4 and up it becomes way too distracting IMO.

What's your personal take on supersampling/oversampling? In my experience it can be a quite useful concept to apply certain filter (eg. DNR, sharpening) not in native source or output resolution, but in one much higher than that (multiples). Although it may sound unintuitive to add an additional resizing step.

Do you expect SO, Gennum and/or ABT to use linear processing in one of their upcoming generations anytime soon?

[scaesare]

Stacey, this is a great thread. Thanks for the examples.

You brewing up anything specific for de-bayering and then down rezi'ng 4K RAW (as opposed to RGB) data in to the 1080p 4:2:0 for HD media use?

You know... in case you happen to end up with some 4K source any time soon.

[MovieSwede]

Interesting how much difference there can be with conversion of the source.

And now when we have 4K sources more and more, this is getting very important.

[sspears]

Steve,

I believe there is always a better way to debayer. One just needs direct access to the RAW bits. e.g. The Cineform SDK allows you to plug-in your own debayer algorithm.

TheLion,

I believe that once you start processing a source, you should stay in the highest quality format until you are done. I would not necessarily resize an image just to perform NR or sharpenss, but I would do it at a higher bitdepth, like 32-bit float. If I am resizing, then I would probably perform that operation before NR or sharpness.

[trbarry]

Originally Posted by sspears

Steve,

I believe there is always a better way to debayer. One just needs direct access to the RAW bits. e.g. The Cineform SDK allows you to plug-in your own debayer algorithm.

TheLion,

I believe that once you start processing a source, you should stay in the highest quality format until you are done. I would not necessarily resize an image just to perform NR or sharpenss, but I would do it at a higher bitdepth, like 32-bit float. If I am resizing, then I would probably perform that operation before NR or sharpness.

Stacy -

I was wondering abut bayer patterns the other day. It seemed to me that RGB (but not YUV) type bayer patterns might need the green filtered down to only the resolutions of the red and blue components to avoid (mild) chroma shift artifacts. Is that true, or a known problem?

- Tom

[MovieSwede]

sspears, why is it better to do NR after resizing?