MPEG2Repair: New Tool for Error Detection (also improves 169time compatibility) - AVS Forum
Forum Jump: 
Reply
 
Thread Tools
post #1 of 426 Old 09-04-2004, 06:01 PM - Thread Starter
AVS Special Member
 
Wizziwig's Avatar
 
Join Date: Jul 2001
Location: SoCal, USA
Posts: 1,336
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 246 Post(s)
Liked: 179
Note: Version 1.0.1.5 is now available (02-15-07).

Please download the attached file or visit:

http://homepages.roadrunner.com/mwilczyn/mpeg2


What's new:

1.0.1.5 / 02-15-07:

Added "bad-edit" detection to show inconsistent telecine flags. Use "IgnoreFieldRepeatFlagWarnings" ini setting to disable.
Added "ReadBufferSize" and "WriteBufferSize" ini settings. Might improve file i/o performance if you're not CPU limited.
Improved performance of initial PID scan on slow wireless networks.
Added "FileWildcard" ini setting to allow custom input/output file extensions.
Added limited H.264 video support. Only scans for Transport/PES header errors.
Removed beta expiration date. Barring any major bugs, this is the final version of MPEG2Repair. Moving on to other projects...



Hello,

I'm pleased to announce the public release of my new Mpeg2 error detection/repair program called "MPEG2Repair". It's designed for testing and repair of HDTV transport streams. Features:

-Error Detection:

Find file errors resulting from signal drop-outs, faulty tapes, encoding errors, etc. Detection code checks transport layer for continuity errors and faulty headers. Code also checks the MPEG2 PES layer for parsing errors (corrupted macroblocks, slices, headers, etc.). All errors are logged with time stamps and file positions so you can easily locate them inside your original recording. Very useful for verifying files prior to permanent archiving.

-Error Repair:

Automatically repair minor Mpeg2 transport and PES errors which may cause problems for certain decoders. Tool will remove erroneous data from the file by either cutting it out or replacing it with valid bits. The goal of repair is to create a fully Mpeg2 compliant file that will not crash, hang, or stall decoders which were not designed to deal with errors. Repair is currently not intended to improve the visual appearance of errors but simply to remove them.

-Improved Decoder Compatibility:

Automatically fix other issues that may affect decoder compatibility. These include partial GOP's, missing headers, incorrect continuity, etc. This feature is especially useful for those of you with old recordings from the 169time system. They should now play smoothly on other decoders besides the JVC 30K deck.

 

mpeg2repair.zip 219.4599609375k . file
Wizziwig is offline  
Sponsored Links
Advertisement
 
post #2 of 426 Old 09-04-2004, 06:49 PM
AVS Special Member
 
h2ofun's Avatar
 
Join Date: Feb 2001
Location: auburn,ca usa
Posts: 3,662
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Mark, GREAT STUFF!!

Dave
h2ofun is offline  
post #3 of 426 Old 09-05-2004, 05:03 PM - Thread Starter
AVS Special Member
 
Wizziwig's Avatar
 
Join Date: Jul 2001
Location: SoCal, USA
Posts: 1,336
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 246 Post(s)
Liked: 179
Thanks for all the email feedback so far. There's definitely a reason I called this an alpha (vs. beta). I know there are lots of bugs yet to be fixed. I figured I would release it anyway since many people seemed interested and it works perfectly for my setup (Dish 6000/169time Hard Drive recording). I never would have found the remaining issues/bugs without getting other people to try it.

To answer a couple of the emails:

1) It's been reported that some virus checkers think there may be a virus in this program. That's a false alarm. The program is packed to produce a smaller exe file. This seems to confuse some virus checkers.

2) Program was compiled with Visual C++ 7.1. If you've never installed any other software that uses that version, you may need some additional DLL files. Here's links to what you need (just unzip them to the same directory as Mpeg2Repair):

mfc71.dll : http://www.dll-files.com/dllindex/dll-files.shtml?mfc71
msvcr71.dll: http://www.dll-files.com/dllindex/dl....shtml?msvcr71

3) The repair feature is intended for minor corruption that causes compatibility problems. It will not fix a recording which is so corrupted that nothing will play it at all. If you have access to a JVC 30K DVHS unit, try playing the original on that. The JVC decoder is very good at dealing with errors. If it can't play your file, it's unlikely that I can do anything to fix it.

4) A large number of continuity errors means that you're losing data somewhere during recording. It could be between the satellite/STB, STB/recorder, or recorder/pc. In any event, large amounts of continuity errors are usually beyond repair. You will need to check your hardware to fix the source of the drop-outs. Also try testing the file with TSReader Lite to make sure the continuity errors are not a false alarm/bug in my program.

In addition to reporting bugs, please post any positive feedback as well. I'm curious if this is useful for anyone else but me. Especially people with Dish6000 recording HBO_HD/SHO_HD.

Thanks.

-Mark
Wizziwig is offline  
post #4 of 426 Old 09-05-2004, 05:40 PM
AVS Special Member
 
Joe Q's Avatar
 
Join Date: Jul 2000
Location: Annapolis,MD
Posts: 3,032
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Dumb question time.

Is this for repairing .tp files or .mpeg files? Ie. Transport Stream or Mpeg2 files

I am asking because I ran it on a .tp file that was created from the HD-TIVO and the program crashed and only generated a 0 byte output file.

The I ran it on a .mpg file that was created from the HD-TIVO and the program crashed trying to find the PID's


Thanks
Joe Q is offline  
post #5 of 426 Old 09-05-2004, 07:30 PM - Thread Starter
AVS Special Member
 
Wizziwig's Avatar
 
Join Date: Jul 2001
Location: SoCal, USA
Posts: 1,336
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 246 Post(s)
Liked: 179
Quote:


Originally posted by Joe Q
Dumb question time.

Is this for repairing .tp files or .mpeg files? Ie. Transport Stream or Mpeg2 files

I am asking because I ran it on a .tp file that was created from the HD-TIVO and the program crashed and only generated a 0 byte output file.

The I ran it on a .mpg file that was created from the HD-TIVO and the program crashed trying to find the PID's


Thanks

Transport Stream files (usually .ts,.tp suffix). It's possible that TIVO is using some proprietary format that I don't yet support. If you like, you can send me a sample (~10 MBytes) to mpeg2repair@adelphia.net. I'll take a look and let you know if I can add support in the future.

On a related note... people should hold off on using this for anything but Dish satellite recordings. I found an issue that affects OTA and possibly DirectTV that I need to address. I'll post a message here when a new version is available.
Wizziwig is offline  
post #6 of 426 Old 09-15-2004, 12:14 PM - Thread Starter
AVS Special Member
 
Wizziwig's Avatar
 
Join Date: Jul 2001
Location: SoCal, USA
Posts: 1,336
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 246 Post(s)
Liked: 179
Just a quick status update...

I'm still working on the next release. I would discourage anyone who's downloaded the first version from using it. There's some serious issues there which make it next to useless for anything but unsplit files from HBO/SHO on Dish. Most other channels and split/segmented files will not work correctly.

The next version keeps getting pushed back as I add new features. I just found out that some OTA stations (like FOX here in LA) and TNT_HD (before Dish started recompressing it) use yet another non-standard format. I'm adding repair support for them now. Current ETA is this weekend.

-Mark
Wizziwig is offline  
post #7 of 426 Old 09-15-2004, 02:03 PM
AVS Club Gold
 
Alan Gouger's Avatar
 
Join Date: Jan 1999
Location: Florida
Posts: 18,728
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 16
Thanks for the updates and your efforts.
Alan Gouger is offline  
post #8 of 426 Old 10-10-2004, 06:00 PM - Thread Starter
AVS Special Member
 
Wizziwig's Avatar
 
Join Date: Jul 2001
Location: SoCal, USA
Posts: 1,336
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 246 Post(s)
Liked: 179
New version now available (see first post in thread). I feel this version is stable enough for general public release. It should now be very reliable at finding errors and dealing with all the major formats. Repair seems good enough for 169time usage. Any other repairs should always be verified to make sure you like the results compared to the original.

Due to lack of time, this will likely be the last release for a while. I hope you find the program useful.

-Mark
Wizziwig is offline  
post #9 of 426 Old 10-10-2004, 10:52 PM - Thread Starter
AVS Special Member
 
Wizziwig's Avatar
 
Join Date: Jul 2001
Location: SoCal, USA
Posts: 1,336
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 246 Post(s)
Liked: 179
I just made a small change (9:45 PM PST) to version 1.0.0.3. It shouldn't affect anything but you may want to download again.

Also, you may be interested in the NullPacketSaver program that's also on the site. It's an alternative to the various null packet removal tools out there (NullPacketStripper, etc.). This one also removes null packets but offers the option to put them back later if needed.

-Mark
Wizziwig is offline  
post #10 of 426 Old 10-13-2004, 03:38 PM
Member
 
Johnny Begood's Avatar
 
Join Date: Feb 2002
Posts: 82
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Thanks a lot Mark for this great software. I've got some recordings that I couldn't read with my system (MyHD) because it was recorded on some other systems. After having them being processed with your software it works just fine.

I have processed 10 movies so far without any problem.

J.
Johnny Begood is offline  
post #11 of 426 Old 10-13-2004, 04:00 PM
AVS Club Gold
 
dahester's Avatar
 
Join Date: Sep 1999
Location: Austin, TX
Posts: 1,106
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 11
Mark,
Another thanks for providing such an excellent tool. I've noticed that even the most stable AVX-1 software revision (08D6) gives streams with occasional corrupted bytes in the MPEG video. I now run every AVX-1 recording through MPEG2Repair before it is transferred to D-VHS. The resulting streams play fine on all of my hardware decoders (MyHD, LG 3410A, Samsung SIRT-165, RCA DTC-169, and JVC 30k). Nice work.

-Dylan

I drink your milkshake! I drink it up!
dahester is online now  
post #12 of 426 Old 10-14-2004, 03:17 PM
rrg
Advanced Member
 
rrg's Avatar
 
Join Date: Jan 2000
Location: NJ
Posts: 795
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:


Originally posted by dahester
I now run every AVX-1 recording through MPEG2Repair before it is transferred to D-VHS. The resulting streams play fine on all of my hardware decoders (MyHD, LG 3410A, Samsung SIRT-165, RCA DTC-169, and JVC 30k).

Dylan: are you using any of the newer AVX1 software releases, like 08D6? Or are you referring to older AVX1 recordings?

169Time has claimed that the new AVX1 software produces recordings that are compatible with a wider variety of decoders than before. But is it in fact necessary to process even newer streams with MPEG2Repair in order to get decoder compatibility?

Ron Gomes
rrg is offline  
post #13 of 426 Old 10-14-2004, 05:13 PM
AVS Club Gold
 
dahester's Avatar
 
Join Date: Sep 1999
Location: Austin, TX
Posts: 1,106
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 11
Ron,
Yes, you should run MPEG2Repair on all your AVX-1 recordings.

I'm using 08D6 with a Dish 6000. I still get the same types of errors as with older software versions, just fewer of them. I've gathered statistics from MPEG2Repair's error log. Older versions of AVX-1 software (pre-08C6) typically have 1 bad frame per 1000. With 08D6, I'm getting 1 bad frame per 10000. Now, bear in mind that I generally start with recordings that have ZERO continuity errors. Recordings with ZERO continuity errors have perfect audio but the MPEG video still has corrupted bytes as an artifact of the AVX-1 software, which are neatly fixed by MPEG2Repair.

I've been on the warpath to banish dropouts, glitches, etc. and MPEG2Repair has been a godsend. I have a pretty strict flow which does not allow much junk to get past me. First I capture everything to the hard drive. I run ec.exe to check for continuity errors. This program can scan a 2 hour movie in 6 to 10 minutes to tell you if you've got errors. I usually split my recording into many files so I can isolate the files which have the errors, watch them in MyHD, and determine their severity. If there are several glitches I often throw out the recording and re-capture it on a later broadcast.

Otherwise I run it through MPEG2Repair and check the error log. Once again, if the error log is suspiciously verbose, I check it out in MyHD to see if the 'fixed' glitches are still severe. After passing the two tests, I transfer the recording to D-VHS and/or archive to DVD-R. Because of all the steps I take, I make fewer recordings now than I used to, but each one I keep is virtually perfect.

If you really want to know how good a capture you have made, you should run your recording through ec.exe to check continuity errors (sound), and then MPEG2Repair (video), and inspect the error logs.

-Dylan

I drink your milkshake! I drink it up!
dahester is online now  
post #14 of 426 Old 10-14-2004, 08:50 PM
rrg
Advanced Member
 
rrg's Avatar
 
Join Date: Jan 2000
Location: NJ
Posts: 795
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Well, that's good news. I hadn't expected that (in particular) older AVX1 streams would be reparable.

It's clearly a labor-intensive process, though.

Have you tried MPEG2Repair on streams originally recorded with the Panasonic TU-DST50 and DST51?

How about streams recorded with the R5000-HD? Are they affected in any way (bad or good) by processing with MPEG2Repair?

Ron Gomes
rrg is offline  
post #15 of 426 Old 10-14-2004, 11:40 PM
AVS Club Gold
 
dahester's Avatar
 
Join Date: Sep 1999
Location: Austin, TX
Posts: 1,106
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 11
I have not tried MPEG2Repair on Panasonic recordings yet. Panny recordings are notorious for being difficult to work with. You can't null strip them without getting into trouble, mainly because the timestamps are really weird. If a program comes along and restamps the PCR, the resultant file will not play back smoothly in most hardware MPEG decoders.
I'm afraid MPEG2Repair might not work well with them.

I don't have an R5000-HD but may get one if they ever go on sale. It's hard to justify a $1000 outlay when the AVX-1 + CapDVHS gets the job done.

-Dylan

I drink your milkshake! I drink it up!
dahester is online now  
post #16 of 426 Old 10-15-2004, 12:12 AM
AVS Special Member
 
mkerdman's Avatar
 
Join Date: Sep 2000
Location: Santa Barbara, CA
Posts: 3,056
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 11
Quote:
Originally posted by dahester
If you really want to know how good a capture you have made, you should run your recording through ec.exe to check continuity errors (sound), and then MPEG2Repair (video), and inspect the error logs.

-Dylan

Dylan


What and where is ec.exe?

Does it fix continuity errors, or, just scan and report them?


EDIT: I got a clue:

"Have you tried balazer's ec.exe utility? " Dylan

Murray Kerdman
mkerdman is offline  
post #17 of 426 Old 10-15-2004, 10:01 AM
AVS Special Member
 
mkerdman's Avatar
 
Join Date: Sep 2000
Location: Santa Barbara, CA
Posts: 3,056
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 11
I think that the missing component in assigning blame for errors are the satellite transmission themselves.

Testing needs to be done to prove what only some people know.

The STB's sold by E* & D* are very good at masking errors, particularly video errors.

While the JVC 30K is also very good, PCI cards like MyHD and software are less so.

Look at the response the R5000 folks are offering for some lost recordings. "Tests have shown that these were always associated with reception glitches and drop-outs."

For various reasons, many of the most active enthusiasts want to use other than D-VHS decks form JVC to playback their HD captures.

The fact remains that since the sat providers are squeezed for bandwidth they are pushing the limits with three channels on a single transponder with various retransmission wizardry and the quality and consistency of the transmissions are suffering for it.

You can see live glitches on SHOHD from E* and many other anecdotal reports of visible errors have occurred in the last 3-4 months.

Still, I do not know how you objectively test the source itself separately from the device used to capture the signal

Murray Kerdman
mkerdman is offline  
post #18 of 426 Old 10-15-2004, 10:51 AM
rrg
Advanced Member
 
rrg's Avatar
 
Join Date: Jan 2000
Location: NJ
Posts: 795
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Though not many of us can take advantage of them, the few Digicipher II C-band HD "master feeds" available to consumers (for HBO, Showtime, Starz, Encore, and Discovery) are probably the best sources to use for captures, since all other providers derive their feeds from these, re-encoding them and re-transmitting, and introducing errors in the process.

The 169Time-modified HDD-200 is currently the only way to record these that I know of, though Nextcom has said that they plan to provide a mod for use with the R5000-HD.

Ron Gomes
rrg is offline  
post #19 of 426 Old 10-15-2004, 12:30 PM
AVS Special Member
 
mkerdman's Avatar
 
Join Date: Sep 2000
Location: Santa Barbara, CA
Posts: 3,056
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 11
Quote:
Originally posted by dahester
Ron,
Yes, you should run MPEG2Repair on all your AVX-1 recordings.

I'm using 08D6 with a Dish 6000. I still get the same types of errors as with older software versions, just fewer of them. I've gathered statistics from MPEG2Repair's error log. Older versions of AVX-1 software (pre-08C6) typically have 1 bad frame per 1000. With 08D6, I'm getting 1 bad frame per 10000. Now, bear in mind that I generally start with recordings that have ZERO continuity errors. Recordings with ZERO continuity errors have perfect audio but the MPEG video still has corrupted bytes as an artifact of the AVX-1 software, which are neatly fixed by MPEG2Repair.

-Dylan

Dylan

I happened to have asked Richard at 169Time about MPEG repair utilities in general, and, he wrote me back as follows:

If a user records and plays on the deck, the deck's built in decoders are robust enough to polish the signal.

When a software based decoder is used that is not error tolerant, a pre-process utility can be applied to mask these errors so they do not cause the decoder to seriously malfunction. That is the function of this software utility being touted here. It masks errors but does not correct them.

A more robust software decoder could also include a masking process and not require this separate step. The pre-playback error masking utility duplicates some of the processing that a software decoder does, and thus it is more logical that the error masking be done by the software decoder at playback time rather than as a separate pass utility, put that on the decoder wish list.

Some hardware based decoders also benefit from this type of pre-playback error masking process.

A robust decoder masks errors in real time at playback as part of the decoding process. Error masking could also become part of the real time process when files are recorded from the transmission. That is the next step toward perfection, but it duplicates some of the process that is typically done for playback decoding.

Bottom line- Even without error masking, the 169time system is making recordings that are equal in quality to the received signal

Richard at 169Time - support@169time.com

Murray Kerdman
mkerdman is offline  
post #20 of 426 Old 10-15-2004, 01:39 PM
AVS Club Gold
 
dahester's Avatar
 
Join Date: Sep 1999
Location: Austin, TX
Posts: 1,106
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 11
I agree that MPEG2Repair isn't correcting the errors per se. Corrupted data is forever lost, but the utility does mask errors to sensitive MPEG decoders very effectively.

I disagree that the 169time system is making recordings that equal the source. I have been lucky enough to make simultaneous recordings of the same channel from the Dish 6000 and a cable box (my cable provider doesn't have its CP flags straight on all channels yet). I have run MPEG2Repair on both and found the cable recordings to have no errors, while the AVX-1 still exhibits errors. Now, I can't rule out the possibility that E* is introducing errors into the stream. But this data makes me suspicious of the AVX-1, especially in light of its history.

BTW reception problems are typically characterized by corrupted video AND audio through lost packets. In all software versions of the AVX-1 that I've tried, when the recording is free of lost packets the audio is perfect, but I still see video errors caught by MPEG2Repair. I have not seen this characterisitic in any other source I've recorded.

-Dylan

I drink your milkshake! I drink it up!
dahester is online now  
post #21 of 426 Old 10-15-2004, 03:04 PM
AVS Special Member
 
mkerdman's Avatar
 
Join Date: Sep 2000
Location: Santa Barbara, CA
Posts: 3,056
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 11
Quote:
Originally posted by dahester
I agree that MPEG2Repair isn't correcting the errors per se. Corrupted data is forever lost, but the utility does mask errors to sensitive MPEG decoders very effectively.

I disagree that the 169time system is making recordings that equal the source. I have been lucky enough to make simultaneous recordings of the same channel from the Dish 6000 and a cable box (my cable provider doesn't have its CP flags straight on all channels yet). I have run MPEG2Repair on both and found the cable recordings to have no errors, while the AVX-1 still exhibits errors. Now, I can't rule out the possibility that E* is introducing errors into the stream. But this data makes me suspicious of the AVX-1, especially in light of its history.

BTW reception problems are typically characterized by corrupted video AND audio through lost packets. In all software versions of the AVX-1 that I've tried, when the recording is free of lost packets the audio is perfect, but I still see video errors caught by MPEG2Repair. I have not seen this characterisitic in any other source I've recorded.

-Dylan

Dylan

I believe your suspicions that the 169Time AVX1 software ADDS errors may be more speculation than fact.

Your post peaked my interest and Richard responded to it as follows:

"Audio and video packets are transmitted separately in time and a video error is 40 times more likely to occur due to the data rates. Errors could also be introduced by Dish's processing. Using the 40X rate and his previously stated error rate of once per 10000 frame (not verified by me), an audio error would occur once in 400,000 frames approximately, or about 4 hours."

"(AVX1 software) Version 08C6 removed all the issues that contributed to stream errors. The only stream errors coming out now are the ones coming in from the satellite receiver."

"Where the goal is to archive at the highest quality, even the results of the repair utility need to be verified since it is a mask routine and not a correction"

It is impossible and really unfair to state unequivocally that the AVX1 software is not a faithful recording that is "equal to the received signal".

You might want to email Richard at support@169time.com to better advance your process.

Murray Kerdman
mkerdman is offline  
post #22 of 426 Old 10-15-2004, 05:12 PM
AVS Club Gold
 
dahester's Avatar
 
Join Date: Sep 1999
Location: Austin, TX
Posts: 1,106
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 11
Murray, thanks for serving as messenger here.

Let me take a different angle on this. I have numerous sources of HD which I have recorded and analyzed: Dish (with AVX-1), cable, OTA, free-to-air satellite, and a considerable library of recordings made with the sorely missed Dish 5000 and modulator. I have considerable experience ripping apart these streams and reconstructing perfect copies by editting the best parts of duplicate copies of a movie in Hi-Def, and re-encoding to produce a perfect version.

The AVX-1 is the only source I've encountered which shows the characteristic that video bytes are corrupted while audio bytes are preserved with no continuity errors appearing in the stream. This has been a hallmark of AVX-1 recordings since day one.

Having said that, perhaps Richard has removed one source of error (i.e. software) only to reveal another source of error which occurs less frequently (i.e. hardware). There are no fewer than four Firewire devices which are needed to make recordings from the AVX-1 to a Windows PC (three is the bare minimum but practically speaking you also need a JVC deck to be able to monitor the AVX-1 output). There is a non-zero probablility that corrupted bytes will occur over Firewire since the data is carried over an isochronous channel, which provides guaranteed bandwidth at the expense of error-free data delivery.

There was a time when I had both a Dish 5000 + modulator and a Dish 6000. At this time unequivocal comparisons could be made between the two systems. Recordings I made with both systems show that the Dish 5000/MyHD recordings are perfect as reported by MPEG2Repair; the AVX-1 recordings, even with no continuity errors, require extensive repairs to the MPEG video while the audio is perfect.

It is possible to show unequivocally whether or not the AVX-1 software is a faithful recording that is 'equal to the received signal.' I propose that someone use an R5000-HD and an AVX-1 system to record the same material. An analysis in MPEG2Repair of the two recordings will verify what I am postulating.

-Dylan

I drink your milkshake! I drink it up!
dahester is online now  
post #23 of 426 Old 10-15-2004, 08:27 PM - Thread Starter
AVS Special Member
 
Wizziwig's Avatar
 
Join Date: Jul 2001
Location: SoCal, USA
Posts: 1,336
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 246 Post(s)
Liked: 179
Hi guys,

Glad to see there is some interest in my program.

I'm not sure if I can go into much detail regarding 169time errors because I signed an NDA with them while helping to debug their new software. But I can explain the mpeg2repair error logs in more detail so maybe you can reach your own conclusions.

Most errors are accompanied by an MBA (Macroblock address) and/or Slice number. An mpeg2 picture is made up of 16x16 pixel squares called macroblocks. Macroblocks are then grouped into horizontal strips/slices that cover a single row of the picture. For satellite, there seems to always be a single slice per macroblock row, but for OTA you often have 2-3 slices per row. To give some real-world examples, here's how things look for a typical 1920x1080i Dish satellite file:

1) Picture is made of 68 slices. Slice number 1 is follow by slice 2 below it, etc. Each horizontal slice contains 120 Macroblocks of 16x16 pixels each for a total of 1920x16 pixels. If you add up the slices, you will see they cover 1920x1088 total pixels. Because the macroblocks must be a multiple of 16 pixels, there is an unused 8 pixels at the bottom of the picture. On some players, it's visible as a small gray border at the bottom of the image.

2) To convert an MBA address to a pixel location, use this formula:

x = 16*(MBA%120); - here '%' stands for mod operator (remainder of a division)
y = 16*(MBA/120); - here you divide and drop the remainder.

For example, MBA=8155, would mean you have an error in a 16x16 macroblock where the top/left pixel is at 1840,1072. Maybe you can find something interesting by looking at the picture locations of 169time errors.


When my tool repairs a file, it basically plays the entire file just like a software mpeg2 decoder would. This allows it to catch all errors in the video steam but makes the process very slow, especially for high resolution, high bitrate files. When the mpeg2 decoder finds a macroblock that it can't decode, it overwrites it with a solid-colored black macroblock. If there are more macroblocks until the next slice, it either skips them (leaving behind whatever was there from the previous frame) or also replaces them with black macroblocks. Skipping is never allowed in I-Pictures so repaired errors on those will show up as black horizontal lines.

A proper mpeg2 decoder (like the JVC 30K) will do the above automatically while playing your file (so repair is not needed). I wrote this tool because most PC decoders designed for DVD playback simply can't cope with errors. They will either crash or skip the entire frame(s) causing noticeable pauses during playback. Due to the way MPEG2 is defined, it's pretty much impossible to really fix errors. Best you can do is try to recover from them. I chose to use black macroblocks to mask the errors because they seemed least noticeable to me (especially when they occur near the black border of the screen). There's other repair options which I may explore in the future (borrowing macroblocks from previous frames, etc.).

You may also like to know that the program does continuity checks on both video and audio packets. So you may skip running the EC utility if you like. If you're recording from cable, it also has the advantage of doing continuity checks in a way that won't report false errors like TSReader, etc.

-Mark
Wizziwig is offline  
post #24 of 426 Old 10-15-2004, 08:38 PM
AVS Special Member
 
dr1394's Avatar
 
Join Date: May 2002
Location: Mizar 5
Posts: 3,174
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 21 Post(s)
Liked: 21
Quote:


For satellite, there seems to always be a single slice per macroblock
row, but for OTA you often have 2-3 slices per row.

Multiple slice headers per row is the mark of a Harris/Lucent "Flexicoder"
HD MPEG-2 encoder widely used for OTA. 1080i will have 3 slices per row
and 720p will have 2. This is a function of the hardware architecture of
the Flexicoder that uses 6 SD encoder chips that execute pretty much
independently of each of each other. Each chip starts it's bitstream with
a slice header.

On all other encoders like the Harmonic MV400 and MV450, the chip
partitions are done vertically, so each chip does entire rows and multiple
slice headers aren't necessary (although you can configure the encoder
to have multiple slice start codes per row to decrease the decoder error
recovery time within a row).

If you really analyze a Flexicoder video stream, you'll see that motion
vectors never cross chip partitions and that automatic I-frame insertion
is done independently on each chip (by coding all intra blocks for that
partition in an overall P-frame).

Ron

HD MPEG-2 Test Patterns http://www.w6rz.net
dr1394 is online now  
post #25 of 426 Old 10-15-2004, 08:45 PM
AVS Special Member
 
robena's Avatar
 
Join Date: Aug 2000
Posts: 1,682
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 12
Wiz,

The new JVC 5U does not like streams where commercials have been removed with HDTVtoMPEG2.

Very often, the transition triggers a very noticeable lip-sync delay.

Could your tool be improved to mask the audio discontinuities introduced by the editing, making the 5U usable with these streams?

Robert
robena is offline  
post #26 of 426 Old 10-15-2004, 08:48 PM
AVS Special Member
 
mkerdman's Avatar
 
Join Date: Sep 2000
Location: Santa Barbara, CA
Posts: 3,056
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 11
Mark

Is it possible to add the MPEGRepair Utility code to the AVX1 recording software to be an on the fly real time process when the files captured on a PC or to a D-VHS deck?

Murray Kerdman
mkerdman is offline  
post #27 of 426 Old 10-15-2004, 08:50 PM
AVS Special Member
 
dr1394's Avatar
 
Join Date: May 2002
Location: Mizar 5
Posts: 3,174
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 21 Post(s)
Liked: 21
Quote:


Originally posted by robena
Wiz,

The new JVC 5U does not like streams where commercials have been removed with HDTVtoMPEG2.

Very often, the transition triggers a very noticeable lip-sync delay.

Could your tool be improved to mask the audio discontinuities introduced by the editing, making the 5U usable with these streams?

Cris Moore is working on this. See this thread:

http://www.avsforum.com/avs-vb/showt...hreadid=452774

Ron

HD MPEG-2 Test Patterns http://www.w6rz.net
dr1394 is online now  
post #28 of 426 Old 10-15-2004, 08:53 PM
AVS Special Member
 
robena's Avatar
 
Join Date: Aug 2000
Posts: 1,682
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 12
Quote:


Originally posted by dr1394
Cris Moore is working on this.

Yep, I know, but it would be nice to be able to (audio) repair already edited streams!

Robert
robena is offline  
post #29 of 426 Old 10-15-2004, 08:56 PM - Thread Starter
AVS Special Member
 
Wizziwig's Avatar
 
Join Date: Jul 2001
Location: SoCal, USA
Posts: 1,336
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 246 Post(s)
Liked: 179
Quote:


Originally posted by dr1394
Multiple slice headers per row is the mark of a Harris/Lucent "Flexicoder"
HD MPEG-2 encoder widely used for OTA. 1080i will have 3 slices per row
and 720p will have 2. This is a function of the hardware architecture of
the Flexicoder that uses 6 SD encoder chips that execute pretty much
independently of each of each other. Each chip starts it's bitstream with
a slice header.

On all other encoders like the Harmonic MV400 and MV450, the chip
partitions are done vertically, so each chip does entire rows and multiple
slice headers aren't necessary.

Ron

Hi Ron,

Good to hear from you again. Without your inspiration, I probably would never have written this program.

I actually wish Dish would use more slices per row since it allows error recovery with less lost information. With Dish, getting an error at the start of a slice means you're losing the entire 1920x16 pixel wide row. On OTA, I often found cases where only a single 40 macroblock slice was bad. In practice, those errors were often not even noticable without using my program (assuming low-motion scenes).

Since you're the mpeg2 master, maybe you can shed some light on the mysterious 0x000001B4 start code. I can't find much information about it other than "sequence error". Where does this come from? Is it inserted by the receiver or at the broadcast station? It seems more common for OTA. Right now, I treat it like any other error but was wondering if there's something special you can do with it (is it followed by any useful data?).

-Mark
Wizziwig is offline  
post #30 of 426 Old 10-15-2004, 09:01 PM - Thread Starter
AVS Special Member
 
Wizziwig's Avatar
 
Join Date: Jul 2001
Location: SoCal, USA
Posts: 1,336
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 246 Post(s)
Liked: 179
Quote:


Originally posted by mkerdman
Mark

Is it possible to add the MPEGRepair Utility code to the AVX1 recording software to be an “on the fly” real time process when the files captured on a PC or to a D-VHS deck?

No. As I explained, the program is essentially an MPEG2 decoder. Try finding a software MPEG2 decoder that can decode HDTV on a 1 Ghz machine (like the AVX) without hardware assistance.

It would simply be too slow to do in real-time unless the AVX was upgraded to something like a 2 Ghz+ machine.

-Mark
Wizziwig is offline  
Reply HDTV Recorders

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off