Originally Posted by coyoteaz
...The encoder first has to detect that the fields are the same, and has to have some level of tolerance in there to account for lossy compression coming from the upstream source. If the tolerance is too low, then some efficiency is lost because not all fields can be matched; if it's too high, then some fields with little motion get incorrectly detected as repeats, which can lead to judder in playback...
You should apply for a job writing the Sharper Image catalog descriptions. I am afraid that this sounds more like a fairy tale, much more made up fantasy than fact. And here is a single fact that completely invalidates that fantasy:
According to "Digital Television Fundamentals" by Michael Robin and Michel Poulin in section 11.4.9 on page 508, there is this very clear statement:
When compression encoding of TV materials is performed, the additional and redundant frames brought by the 3:2 pull-down process must be removed from the video signal...Source identification is important for the source-adaptive capability of all devices included in the DTV production and delivery chain (...receivers, etc.)...In ATSC program delivery, film-originating materials are sent with a program identifier included in the program stream.
(I won't bore you with the other three sources I have that state very similar facts regarding the rules by which MPEG program streams are constructed).
That statement of fact would clearly indicate that there is no need for the encoder to "detect" anything regarding the comparison of successive fields, simply because the encoder has already been instructed in packet-header metadata that the content is originally from film.
And of course anything originally from film is 24 fps, which means for it to get to a frame rate of 30 fps 3:2 pulldown is necessary, which, since in that process each field is duplicated five times in a row and then every other field is discarded, by definition means that there are and must be successive duplicated fields arranged in 3:2 pulldown order. And there really is no grey area or wiggle room here for vague descriptions to wriggle out of. This is pretty black and white, and virtually indisputable.
So, under those circumstances (the circumstances that apply in each and every case of ATSC broadcasting of original-film material) it would not matter whether successive fields are 100% or 90% or even less identical, because the very simple instructions in such a case to the encoder are, for each and every successive 60 fields of content, to discard completely
fields 2, 3, 5, 7, 8, 10, 12, 13, 15, 17, 18, 20, 22, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, and 60, and to transmit only
fields 1, 4, 6, 9, 11, 14, 16, 19, 21, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, and 59. Or, of 60 consecutive fields, completely discard
these particular 36 duplicate fields, and send only
these other remaining original 24 fields, each of which represents a frame of the original film. You will notice this not a condition to where the modifier "sometimes" might or could possibly be attached. On the contrary, this is a basic tenet, an unbreakable rule of MPEG encoding for ATSC delivery.
This also instructs the decoder in every STB or tuner through a like mechanism to duplicate the received frames as necessary and interleave copies of them into the transmitted frames to recreate the original pulldown and 60-field rate just as if it were there all along, which every licensed MPEG-2 decoder can do very easily and completely losslessly.
IOW, "detection" of film content is not by comparison of successive frames to see how similar they either are or are not, but by simply obeying the order given in the metadata flag that since it has been declared to be original film content, to obey certain instructions, the above instructions, regarding what gets transmitted and what does not. That is a very simple task, and path degradation or any other variables you can dream up and throw at it will not affect it whatsoever.
The MPEG spec is very specific regarding what it contains and what it does as far as the algorithm itself goes. Rules can't simply be ignored or broken on a whim.The decoding process must always be an exact mirror reverse of the encoding process, for all intents and purposes, and you can't change one of those two independently. The spec is written to account for each of the tools in the toolkit, and the decoder is designed to follow the instructions and parameters set for each of those tools. While the particular degree of the parameters of the tools are set by the compressionist, this does not allow ignoring the actual rules set by SMPTE in conforming to MPEG-2. "Film mode" is not optional.
That is not to say MPEG-2 is not without issues. While the spec is extremely specific about what the algorithm must include, and that you must accomplish certain things to get from point A to point B, exactly how
to accomplish the goals set in the spec, is completely open, left completely to the encoder and decoder manufacturers. But the rules themselves must still be followed. They are rules, after all.
That is the best compromise of having fixed rules but not carving how they are interpreted in stone. This allows order but also allows technology to flourish in finding new and better ways to implement the tools in the MPEG toolkit over time, and the improvements in the last 10 years in HD are a testament to that.
But the tradeoff for not being explicit in exactly how you get from A to B is that sometimes decoder manufacturers can get off the rails a little bit and there can be local issues of incompatibility which need certain workarounds applied to keep everyone happy.
That may explain murky reports like the one you have described; having lived in that world since broadcasting first embraced MPEG, I can recount numerous horror stories of incompatibility that are very similar. But none of them are due to the rules for MPEG not being written immutably, and all of them stem from poor interpretation and execution of the methods used to interpret those rules.