MPEG2Repair: New Tool for Error Detection (also improves 169time compatibility)

Note: Version 1.0.1.5 is now available (02-15-07).

Please download the attached file or visit:

http://homepages.roadrunner.com/mwilczyn/mpeg2


What's new:

1.0.1.5 / 02-15-07:

Added "bad-edit" detection to show inconsistent telecine flags. Use "IgnoreFieldRepeatFlagWarnings" ini setting to disable.
Added "ReadBufferSize" and "WriteBufferSize" ini settings. Might improve file i/o performance if you're not CPU limited.
Improved performance of initial PID scan on slow wireless networks.
Added "FileWildcard" ini setting to allow custom input/output file extensions.
Added limited H.264 video support. Only scans for Transport/PES header errors.
Removed beta expiration date. Barring any major bugs, this is the final version of MPEG2Repair. Moving on to other projects...



Hello,

I'm pleased to announce the public release of my new Mpeg2 error detection/repair program called "MPEG2Repair". It's designed for testing and repair of HDTV transport streams. Features:

-Error Detection:

Find file errors resulting from signal drop-outs, faulty tapes, encoding errors, etc. Detection code checks transport layer for continuity errors and faulty headers. Code also checks the MPEG2 PES layer for parsing errors (corrupted macroblocks, slices, headers, etc.). All errors are logged with time stamps and file positions so you can easily locate them inside your original recording. Very useful for verifying files prior to permanent archiving.

-Error Repair:

Automatically repair minor Mpeg2 transport and PES errors which may cause problems for certain decoders. Tool will remove erroneous data from the file by either cutting it out or replacing it with valid bits. The goal of repair is to create a fully Mpeg2 compliant file that will not crash, hang, or stall decoders which were not designed to deal with errors. Repair is currently not intended to improve the visual appearance of errors but simply to remove them.

-Improved Decoder Compatibility:

Automatically fix other issues that may affect decoder compatibility. These include partial GOP's, missing headers, incorrect continuity, etc. This feature is especially useful for those of you with old recordings from the 169time system. They should now play smoothly on other decoders besides the JVC 30K deck.

 

mpeg2repair.zip 219.4599609375k . file

Thanks for all the email feedback so far. There's definitely a reason I called this an alpha (vs. beta). I know there are lots of bugs yet to be fixed. I figured I would release it anyway since many people seemed interested and it works perfectly for my setup (Dish 6000/169time Hard Drive recording). I never would have found the remaining issues/bugs without getting other people to try it.

To answer a couple of the emails:

1) It's been reported that some virus checkers think there may be a virus in this program. That's a false alarm. The program is packed to produce a smaller exe file. This seems to confuse some virus checkers.

2) Program was compiled with Visual C++ 7.1. If you've never installed any other software that uses that version, you may need some additional DLL files. Here's links to what you need (just unzip them to the same directory as Mpeg2Repair):

mfc71.dll : http://www.dll-files.com/dllindex/dll-files.shtml?mfc71
msvcr71.dll: http://www.dll-files.com/dllindex/dl....shtml?msvcr71

3) The repair feature is intended for minor corruption that causes compatibility problems. It will not fix a recording which is so corrupted that nothing will play it at all. If you have access to a JVC 30K DVHS unit, try playing the original on that. The JVC decoder is very good at dealing with errors. If it can't play your file, it's unlikely that I can do anything to fix it.

4) A large number of continuity errors means that you're losing data somewhere during recording. It could be between the satellite/STB, STB/recorder, or recorder/pc. In any event, large amounts of continuity errors are usually beyond repair. You will need to check your hardware to fix the source of the drop-outs. Also try testing the file with TSReader Lite to make sure the continuity errors are not a false alarm/bug in my program.

In addition to reporting bugs, please post any positive feedback as well. I'm curious if this is useful for anyone else but me. Especially people with Dish6000 recording HBO_HD/SHO_HD.

Thanks.

-Mark

Transport Stream files (usually .ts,.tp suffix). It's possible that TIVO is using some proprietary format that I don't yet support. If you like, you can send me a sample (~10 MBytes) to mpeg2repair@adelphia.net. I'll take a look and let you know if I can add support in the future.

On a related note... people should hold off on using this for anything but Dish satellite recordings. I found an issue that affects OTA and possibly DirectTV that I need to address. I'll post a message here when a new version is available.

Just a quick status update...

I'm still working on the next release. I would discourage anyone who's downloaded the first version from using it. There's some serious issues there which make it next to useless for anything but unsplit files from HBO/SHO on Dish. Most other channels and split/segmented files will not work correctly.

The next version keeps getting pushed back as I add new features. I just found out that some OTA stations (like FOX here in LA) and TNT_HD (before Dish started recompressing it) use yet another non-standard format. I'm adding repair support for them now. Current ETA is this weekend.

-Mark

New version now available (see first post in thread). I feel this version is stable enough for general public release. It should now be very reliable at finding errors and dealing with all the major formats. Repair seems good enough for 169time usage. Any other repairs should always be verified to make sure you like the results compared to the original.

Due to lack of time, this will likely be the last release for a while. I hope you find the program useful.

-Mark

I just made a small change (9:45 PM PST) to version 1.0.0.3. It shouldn't affect anything but you may want to download again.

Also, you may be interested in the NullPacketSaver program that's also on the site. It's an alternative to the various null packet removal tools out there (NullPacketStripper, etc.). This one also removes null packets but offers the option to put them back later if needed.

-Mark

Hi guys,

Glad to see there is some interest in my program.

I'm not sure if I can go into much detail regarding 169time errors because I signed an NDA with them while helping to debug their new software. But I can explain the mpeg2repair error logs in more detail so maybe you can reach your own conclusions.

Most errors are accompanied by an MBA (Macroblock address) and/or Slice number. An mpeg2 picture is made up of 16x16 pixel squares called macroblocks. Macroblocks are then grouped into horizontal strips/slices that cover a single row of the picture. For satellite, there seems to always be a single slice per macroblock row, but for OTA you often have 2-3 slices per row. To give some real-world examples, here's how things look for a typical 1920x1080i Dish satellite file:

1) Picture is made of 68 slices. Slice number 1 is follow by slice 2 below it, etc. Each horizontal slice contains 120 Macroblocks of 16x16 pixels each for a total of 1920x16 pixels. If you add up the slices, you will see they cover 1920x1088 total pixels. Because the macroblocks must be a multiple of 16 pixels, there is an unused 8 pixels at the bottom of the picture. On some players, it's visible as a small gray border at the bottom of the image.

2) To convert an MBA address to a pixel location, use this formula:

x = 16*(MBA%120); - here '%' stands for mod operator (remainder of a division)
y = 16*(MBA/120); - here you divide and drop the remainder.

For example, MBA=8155, would mean you have an error in a 16x16 macroblock where the top/left pixel is at 1840,1072. Maybe you can find something interesting by looking at the picture locations of 169time errors.


When my tool repairs a file, it basically plays the entire file just like a software mpeg2 decoder would. This allows it to catch all errors in the video steam but makes the process very slow, especially for high resolution, high bitrate files. When the mpeg2 decoder finds a macroblock that it can't decode, it overwrites it with a solid-colored black macroblock. If there are more macroblocks until the next slice, it either skips them (leaving behind whatever was there from the previous frame) or also replaces them with black macroblocks. Skipping is never allowed in I-Pictures so repaired errors on those will show up as black horizontal lines.

A proper mpeg2 decoder (like the JVC 30K) will do the above automatically while playing your file (so repair is not needed). I wrote this tool because most PC decoders designed for DVD playback simply can't cope with errors. They will either crash or skip the entire frame(s) causing noticeable pauses during playback. Due to the way MPEG2 is defined, it's pretty much impossible to really fix errors. Best you can do is try to recover from them. I chose to use black macroblocks to mask the errors because they seemed least noticeable to me (especially when they occur near the black border of the screen). There's other repair options which I may explore in the future (borrowing macroblocks from previous frames, etc.).

You may also like to know that the program does continuity checks on both video and audio packets. So you may skip running the EC utility if you like. If you're recording from cable, it also has the advantage of doing continuity checks in a way that won't report false errors like TSReader, etc.

-Mark

Hi Ron,

Good to hear from you again. Without your inspiration, I probably would never have written this program.

I actually wish Dish would use more slices per row since it allows error recovery with less lost information. With Dish, getting an error at the start of a slice means you're losing the entire 1920x16 pixel wide row. On OTA, I often found cases where only a single 40 macroblock slice was bad. In practice, those errors were often not even noticable without using my program (assuming low-motion scenes).

Since you're the mpeg2 master, maybe you can shed some light on the mysterious 0x000001B4 start code. I can't find much information about it other than "sequence error". Where does this come from? Is it inserted by the receiver or at the broadcast station? It seems more common for OTA. Right now, I treat it like any other error but was wondering if there's something special you can do with it (is it followed by any useful data?).

-Mark

No. As I explained, the program is essentially an MPEG2 decoder. Try finding a software MPEG2 decoder that can decode HDTV on a 1 Ghz machine (like the AVX) without hardware assistance.

It would simply be too slow to do in real-time unless the AVX was upgraded to something like a 2 Ghz+ machine.

-Mark