or Connect
AVS › AVS Forum › HDTV › HDTV Recorders › MPEG2Repair: New Tool for Error Detection (also improves 169time compatibility)
New Posts  All Forums:Forum Nav:

MPEG2Repair: New Tool for Error Detection (also improves 169time compatibility) - Page 3

post #61 of 426
Quote:


Originally posted by Joe Q
Mark,
HOWEVER, MYHD is obviously having trouble with the file because the video breaks up throughout the file on playback.

Thanks,

Joe

I've used Mark's utility on 10 movies coming from various source so far. All of them were not watchable with MyHD before (no sound at all and lot of glitches...). All of them are now watchable with still some video glitches (but a lot less) but no audio problem.

I've tried several version of MyHD and I figured out that the best one to play unperfect TS streams is version 1.62.3. The version 1.63 lost audio almost on each glitch (you need to pause and restart to have it back). And the versions before produce more audio glitches.

J.

See there : http://www.avsforum.com/avs-vb/showt...hreadid=455882
post #62 of 426
Mark,

Maybe it's gonna sound futile but it would be cool if your utility could be minimized. I use to process several files at the same time and I just can get rid of all the windows...

J.
post #63 of 426
Mark,
I had not thought about the command line returning right after launching the GUI. I'm not as gifted a scripter as some (batch files are as far as I go) so I think a text mode version would be nicer, if it's not too much trouble to create. BTW is there any way for the GUI version to keep control of the command prompt until it's done?

I have an update after using this tool on many AVX-1 recordings. I found another bug in my "08D6" recordings which I could not identify the source of until now. The symptom is when I play my recordings (trasferred to D-VHS) on the JVC 30k deck, the JVC decoder shows occasional loss of audio and video. It's completely repeatable by rewinding the tape and replaying the section where the dropout occured.

I was initially suspicious of MPEG2Repair, but last night I made two copies of a movie, one pre-repair and one post-repair. It turns out the dropouts occur in the exact same places on both tapes. (BTW the recording was Bad Boys II, which was the cleanest recording I've made from the AVX-1 with only 3 frames that had corrupted data, and zero continuity errors). Now here's where it gets interesting: the suspicious parts of both recordings play back perfectly on ALL other equipment at my disposal (MyHD, Mits 1100U + Samsung T165, and Mits 2000U + LG LST-3410A). I did notice a single frame stutter on the Samsung shortly after the positions where the dropouts were happening on the JVC 30k, but no loss of audio.
BTW the JVC 30k's firewire output drops out simultaneously with the internal decoder dropping out.

Bottom line: This may mean the AVX-1 is subtly, erroneously stamping the PTSes and/or PCR in such a way that the JVC decoder can't handle them, but other decoders can. Another possibility is a slight buffer underrun or overrun in either demuxed audio or video output which causes the 30k some trouble but not the other decoders. Some people were reporting similar problems with the JVC 30k and 08E6 firmware; I think I'm seeing the same issue with 08D6.

-Dylan
post #64 of 426
Quote:


Originally posted by Johnny Begood
I've tried several version of MyHD and I figured out that the best one to play unperfect TS streams is version 1.62.3. The version 1.63 lost audio almost on each glitch (you need to pause and restart to have it back). And the versions before produce more audio glitches.



Thanks for the tip - I will give it a shot.

There is lot of MYHD bashing going on in a well known forum dedicated to the TIVO but for High Def shows, MYHD is the only choice for playing back files made from the HDTIVO.

Joe
post #65 of 426
Thread Starter 
Quote:


Originally posted by Johnny Begood
Mark,

Maybe it's gonna sound futile but it would be cool if your utility could be minimized. I use to process several files at the same time and I just can get rid of all the windows...

J.

Wait for next version. I already fixed this along with many other user-interface improvements.

Quote:


Originally posted by dahester

Bottom line: This may mean the AVX-1 is subtly, erroneously stamping the PTSes and/or PCR in such a way that the JVC decoder can't handle them, but other decoders can. Another possibility is a slight buffer underrun or overrun in either demuxed audio or video output which causes the 30k some trouble but not the other decoders. Some people were reporting similar problems with the JVC 30k and 08E6 firmware; I think I'm seeing the same issue with 08D6.

-Dylan

For audio, I think the buffer under/over run is most likely the problem. I've seen this happen frequently with recordings from DVB cards. I've also had audio drop-outs on older AVX software when running on AVX machines slower than ~P3-800. You might be able to fix the audio drop-outs by shifting the audio packets with Balazer's TStoATSC3. I used to have a similar shifting feature in MP2R but disabled it because it needed more work.


Quote:


Originally posted by Joe Q
Mark,

Do you feel that your program is ready to be used with other data instead of just the 169time system?


I am asking because I ran your program on a 27 Gbyte file that was converted from the HDTIVO (Directv High Def receiver/PVR).

it passed with flying colors ie.It found no errors and the repaired file was the same exact size as the original.

HOWEVER, MYHD is obviously having trouble with the file because the video breaks up throughout the file on playback.

Thanks,

Joe

I don't know the source of all the samples that people have sent me but it's possible that I've never tried anything from the HDTivo. If you like, you can send me a small clip (<10 MB) to mpeg2repair@adelphia.net. Did the error log summary contain an actual frame count and sequence time duration? If not, then my tool probably didn't find any mpeg2 picture data in the file (hence no errors).

If I can't figure it out, maybe Ron (dr1394) might have a clue. He's the true mpeg2 master - I'm just a mere apprentice.

-Mark
post #66 of 426
Quote:


Originally posted by Joe Q
Thanks for the tip - I will give it a shot.

There is lot of MYHD bashing going on in a well known forum dedicated to the TIVO but for High Def shows, MYHD is the only choice for playing back files made from the HDTIVO.

Joe

Actually there are quite a few software-only players that can handle HD playback, including HD from a tivo.
post #67 of 426
I tried this software, thanks for making it. But it says it can't find PID values. Does that mean it's not actually recording anything? Because the files are reasonably sized, 30MB for 30 seconds or so, so I wonder what it is recording. Anyway, I'll make a smaller one and email it to you. Thanks again.
post #68 of 426
Quote:


Originally posted by hokkemirin
I tried this software, thanks for making it. But it says it can't find PID values. Does that mean it's not actually recording anything? Because the files are reasonably sized, 30MB for 30 seconds or so, so I wonder what it is recording. Anyway, I'll make a smaller one and email it to you. Thanks again.

Blank TS files are typical when you try to dump 5C copy protected material.
post #69 of 426
Good program. Only when I did one 30 show the error report was 20 megs long and the file was 1 gig smaller because it cut out the 2nd channel.

When I did other 60 min show, it did not cut out the other channel and file was the same size and the second channel was not seen in VLC.

Maybe because I am using WINHD and it encrips the streem.

Why so amny errors and where is the other channel?
post #70 of 426
Thread Starter 
Quote:


Originally posted by GoTimothy
Good program. Only when I did one 30 show the error report was 20 megs long and the file was 1 gig smaller because it cut out the 2nd channel.

When I did other 60 min show, it did not cut out the other channel and file was the same size and the second channel was not seen in VLC.

Maybe because I am using WINHD and it encrips the streem.

Why so amny errors and where is the other channel?

If the data is encrypted, then there will be no way for the program to make any sense of it. That could explain why you're getting a huge error log.

The program only leaves a single program/channel in a repaired file. This was done because it's what most users want and because it improves compatibility. Some players get confused by multiple programs or end up selecting the wrong one. If you wish to leave a different program in the repaired file, enter its number into the "Program" edit box. You can get the number from a program like TSReader Lite. Next version will no longer require TSReader since I added a drop-down list of available programs.

-Mark
post #71 of 426
Quote:


Originally posted by Wizziwig


The program only leaves a single program/channel in a repaired file. This was done because it's what most users want and because it improves compatibility. Some players get confused by multiple programs or end up selecting the wrong one. If you wish to leave a different program in the repaired file, enter its number into the "Program" edit box. You can get the number from a program like TSReader Lite. Next version will no longer require TSReader since I added a drop-down list of available programs.

-Mark

Mark, when you remove the other programs/channels, do you convert that data to null packets and preserve the ATSC 19.3Mb/sec data rate?

DonP
post #72 of 426
Quote:


Originally posted by Wizziwig

For audio, I think the buffer under/over run is most likely the problem. I've seen this happen frequently with recordings from DVB cards. I've also had audio drop-outs on older AVX software when running on AVX machines slower than ~P3-800. You might be able to fix the audio drop-outs by shifting the audio packets with Balazer's TStoATSC3. I used to have a similar shifting feature in MP2R but disabled it because it needed more work.

Unfortunately the audio and video drop out at the same time on the JVC 30k (I verified it's a problem on two different 30k decks). I tried TStoATSC3 and it doesn't clear up the problem. Since it only affects the JVC 30k I can live with it. Ironic though, since the JVCs were the most compatible with older versions of AVX-1 software.

I was going to send you a clip, but you have the same setup I do (an AVX-1 with a Dish 6000). Make sure to watch your movies all the way through. It's hard to catch just by sampling.

-Dylan
post #73 of 426
Mark,

As I was using your utility on a movie, I was wondering why the output file was smaller (a lot, around 35%) than the input file. I discovered that the file output by MPEG2Repair was cleaned of null packets. Is it also a function of this utility ???

As you mentioned that you've got something else to do this job, I was surprised.

I've also tried on an already null packet cleaned file. The original file size is 4,4 GO and the output file is 3,9. Where are the 500 MO missing ?

J.
post #74 of 426
J.,
I've seen this before also. If your recording originally came from the Dish 5000 modulator, it will have '0x1FEF' null packets rather than the standard '0x1FFF' nulls. Mark's program leaves in the 0x1FFF nulls but will strip out everything else except the main program. IMO it's kind of a problem since recordings from the Panasonic TU-DST5X tuner have this strange property that stripping out nulls can make the resulting file have very jerky playback on many decoders. While the jerky playback can happen on other null stripped recordings, smooth playback can usually be restored by reinserting the nulls and restamping the PCR with TStoATSC(3). This is NOT the case with Panny recordings; once the nulls are gone, you may never be able to restore smooth playback.

-Dylan
post #75 of 426
Thread Starter 
Hmm.... unless you enable nullpacket stripping via the INI file (it's disabled by default), the program should never drop any packets from the transport without replacing them with null packets. It should work regardless of whether the original null packets use PID 0x1FFF or 0x1FEF. Any additional null packets added by my program will always use PID 0x1FFF. This is done so that the original transport bitrate is maintained. Nobody should ever enable null packet stripping unless they are 100% certain that their decoder will function without them. Technically speaking, a null packet stripped TS file is broken and there's no guarantee of it working properly.

If you don't have the INI change and you're still losing packets, then there's a bug. I'll try to double check this on my end. Also make sure you're using the latest version (1.0.0.4) by checking the exe file properties.

-Mark
post #76 of 426
Thread Starter 
Quote:


Originally posted by dahester
Unfortunately the audio and video drop out at the same time on the JVC 30k (I verified it's a problem on two different 30k decks). I tried TStoATSC3 and it doesn't clear up the problem. Since it only affects the JVC 30k I can live with it. Ironic though, since the JVCs were the most compatible with older versions of AVX-1 software.

I was going to send you a clip, but you have the same setup I do (an AVX-1 with a Dish 6000). Make sure to watch your movies all the way through. It's hard to catch just by sampling.

-Dylan

Just to review, do these drop-outs always occur at the same spots when watching multiple times on the JVC 30K? Does my program find any errors in those spots? Is it isolated to a particular channel? These days, I only record HDNet from Dish and have not seen this problem. I get HBO/SHO from another source because I don't like Dish re-compression of those channels.
post #77 of 426
Wizziwig, just to clarify your last statement, the FAQ to MPEG2Repair states the following:
Quote:


5) When repairing the stream, data is often added/removed. As a result, if your original file was constant bitrate (i.e 19.4 Mbps), it may no longer be constant after repair. If your decoder requires constant bitrate files, you may have to remux them or try Balazer's TStoATSC utility to get it back to 19.4 Mbps (http://www.cs.rochester.edu/~balazer/atsc/).

Does this statement still apply to version 1.0.0.4? If one does not enable null packet stripping, does the bitrate stay constant after the file is repaired, so that the TStoATSC utility is not required?
post #78 of 426
Mark,

I was using version 1.0.0.2. The INI file was the following :

StripNullPackets=0
StripEarlyAudioPackets=1
IgnorePCRIntervalWarnings=1
IgnoreContinuityWarnings=0
IgnoreRepeatedPacketWarnings=0
FixMacroblockErrors=1

I've just tried on the same movie with version 1.0.0.4 with the following INI file :

StripNullPackets=0
StripEarlyAudioPackets=0
FixMacroblockErrors=1
IgnoreContinuityWarnings=0
IgnorePCRIntervalWarnings=1
IgnoreRepeatedPacketWarnings=1
IgnoreAudioDelayWarnings=1

And the winner is...v1.0.0.4 which does not seem to remove anything more than errors. The resulting file is almost the same size than the original one.

I have just watched a movie processed by v1.0.0.2. It works really great. I'm now using v1.0.0.4. As my system (MyHD) handles correctly null packet removed streams, if I want MPEG2Repair to repair AND remove null packets, do I just have to change the INI file corresponding line to 1 ?

Thanks,

J.
post #79 of 426
Thread Starter 
Quote:


Originally posted by stjr
Wizziwig, just to clarify your last statement, the FAQ to MPEG2Repair states the followingoes this statement still apply to version 1.0.0.4? If one does not enable null packet stripping, does the bitrate stay constant after the file is repaired, so that the TStoATSC utility is not required?

I made some refinements since the first version (where I added that comment to the FAQ) so that repaired files are now virtually identical in bitrate to the original. In my experience, 99% of the repairs now end up using space that was originally occupied by the corrupted bytes. So there will not be any increase/decrease in transport packets. There are rare occasions where I need to insert a packet or two but those are extremely rare and should not affect bitrate by a significant amount. I'll try to update the FAQ to reflect those changes.

Just to be clear, if you're seeing a drop in file size as a result of my program, there are only a few possibilities:

1) Your file had a large amount of data before the first mpeg2 sequence header. Since such data can't be decoded/displayed by any decoder, I always remove it from the transport. This unused data at the start of your file is not replaced by null packets. Typically this only amounts to < 1MByte of data.

2) Your file contains non-transport data. The error log would report packets without sync-bytes. These can't be played/used by any decoder so they are removed without replacement.

3) Bug in my program. Please contact me if you find an example.

-Mark
post #80 of 426
Thread Starter 
Quote:


Originally posted by Johnny Begood
Mark,

I was using version 1.0.0.2. The INI file was the following :

StripNullPackets=0
StripEarlyAudioPackets=1
IgnorePCRIntervalWarnings=1
IgnoreContinuityWarnings=0
IgnoreRepeatedPacketWarnings=0
FixMacroblockErrors=1

I've just tried on the same movie with version 1.0.0.4 with the following INI file :

StripNullPackets=0
StripEarlyAudioPackets=0
FixMacroblockErrors=1
IgnoreContinuityWarnings=0
IgnorePCRIntervalWarnings=1
IgnoreRepeatedPacketWarnings=1
IgnoreAudioDelayWarnings=1

And the winner is...v1.0.0.4 which does not seem to remove anything more than errors. The resulting file is almost the same size than the original one.

I have just watched a movie processed by v1.0.0.2. It works really great. I'm now using v1.0.0.4. As my system (MyHD) handles correctly null packet removed streams, if I want MPEG2Repair to repair AND remove null packets, do I just have to change the INI file corresponding line to 1 ?

Thanks,

J.

Glad to hear it wasn't a bug after all.

You could do that... but I recommend using my NullPacketSaver utility instead. It will also remove the null packets but store a record of their positions in a backup file. You can later use this backup file to put the null packets back in case you change players, cards, etc. The backup file is relatively small and can be zipped to make it even smaller.

-Mark
post #81 of 426
Quote:


Originally posted by Wizziwig
I made some refinements since the first version (where I added that comment to the FAQ) so that repaired files are now virtually identical in bitrate to the original.

Thanks for the clarification, Mark. The first time I processed a file with your application, the file size stayed the same, so I wondered what was going on. I prefer to keep the bitrate constant for decoder compatibility. I've had good results so far, so now I'll give it a real workout. Thanks for the application.
post #82 of 426
Quote:
Originally posted by Wizziwig
It would be relatively easy to make a command-line version since that's how it started originally. It also used to process mpeg2 elementary streams but I dropped that support because it didn't seem useful for HD (which usually comes in TS files).

Do you actually need a text-mode version of the tool, or can I simply add command-line arguments for input/output paths to the existing GUI version? The only thing missing from a GUI version would be the error return codes that one normally tests in batch files.

Hi Mark.

I'm a Mac / Unix user, and as such live in a veritably wasteland when it comes to TS repair programs. I actually sought this thread out because it sounded like something that could be run from a command line, and lo-and-behold, other users have made this very request!

I am wondering about the possibility of a commond-line version port to the UNIX layer of Mac OS X (Darwin), and to Linux? Both use gcc as their core compiler and porting to one is a huge step in getting it to the other (though there are differences in "stat" and other low level I/O calls).

For example, dr1394's "xport.c" program actually compiled without modification under darwin, and I was able to port TStoATSC without much trouble.

Obviously your program has a Windows interface, and if it's tightly coupled with the processing module, this won't be possible. However, if the file I/O and TS processing can be "broken out," I for one would beg you for a port. I'm hoping your reference to a "text mode" means that these modules can be used independantly.

Those are big "ifs" but in case it's possible, I would be happy to actually do the port (I have access to Debian Linux as well as Mac OS X / Darwin). I would willingly sign an NDA or whatever... though I know very little about MPEG2 TS format and probably wouldn't be able to disclose anything if I tried.

-Pie
post #83 of 426
Thread Starter 
Quote:
Originally posted by EatingPie
Hi Mark.

I'm a Mac / Unix user, and as such live in a veritably wasteland when it comes to TS repair programs. I actually sought this thread out because it sounded like something that could be run from a command line, and lo-and-behold, other users have made this very request!

I am wondering about the possibility of a commond-line version port to the UNIX layer of Mac OS X (Darwin), and to Linux? Both use gcc as their core compiler and porting to one is a huge step in getting it to the other (though there are differences in "stat" and other low level I/O calls).

For example, dr1394's "xport.c" program actually compiled without modification under darwin, and I was able to port TStoATSC without much trouble.

Obviously your program has a Windows interface, and if it's tightly coupled with the processing module, this won't be possible. However, if the file I/O and TS processing can be "broken out," I for one would beg you for a port. I'm hoping your reference to a "text mode" means that these modules can be used independantly.

Those are big "ifs" but in case it's possible, I would be happy to actually do the port (I have access to Debian Linux as well as Mac OS X / Darwin). I would willingly sign an NDA or whatever... though I know very little about MPEG2 TS format and probably wouldn't be able to disclose anything if I tried.

-Pie

After further consideration, doing a text-mode version is probably not going to happen. It's not so much because it's difficult but because it would be a pain in the future. This tool is already taking up most of my free time and maintaining 2 versions would make things even more difficult. I also plan to greatly expand this tool in the future by including some editing functionality. This wouldn't mesh well with text-mode.

Isn't there some kind of Windows emulator for Linux? (Wine?). There should be something similar for Mac. This tool uses pretty basic Windows functions so it should run okay on an emulator.

-Mark
post #84 of 426
I am puzzled as to why a 169Time recording that appears to play fine in MyHD twice caused the attached message from MPEGRepair on the same recording of Down with Love both times within the first few minutes.

All my other 169Time files ran through just fine.
LL
post #85 of 426
Thread Starter 
Quote:
Originally posted by mkerdman
I am puzzled as to why a 169Time recording that appears to play fine in MyHD twice caused the attached message from MPEGRepair on the same recording of Down with Love both times within the first few minutes.

All my other 169Time files ran through just fine.

This is a generic error that occurs when there are too many sequential corrupted frames in a row. In the current version it's set to about 8 MBytes worth of data. When you previously watched this program, was it from the exact same file that you ran through my tool? If not, maybe it got corrupted while being transfered to the PC. Try watching the original unrepaired file that you have on the PC hard drive to see if it has an error in the last position indicated by the log.

In any event, please send me your error log at mpeg2repair@adelphia.net

Thanks.
post #86 of 426
Thread Starter 
Posted new versions of MPEG2Repair (1.0.0.5) and NullPacketSaver (1.02). These updates are to improve user-friendliness/interface. No major changes to functionality. Please keep sending ideas and bug reports for future versions.

-Mark
post #87 of 426
Quote:


Originally posted by Wizziwig
This is a generic error that occurs when there are too many sequential corrupted frames in a row. In the current version it's set to about 8 MBytes worth of data. When you previously watched this program, was it from the exact same file that you ran through my tool? If not, maybe it got corrupted while being transfered to the PC. Try watching the original unrepaired file that you have on the PC hard drive to see if it has an error in the last position indicated by the log.

In any event, please send me your error log at mpeg2repair@adelphia.net

Thanks.

The files run through the tool were never on a D-VHS deck- only the PC.
post #88 of 426
Thread Starter 
Quote:


Originally posted by mkerdman
The files run through the tool were never on a D-VHS deck- only the PC.

Well, if you would like me to look at this issue, I would need a sample of the file. If you have a fast internet connection, you can email me samples < 10 MB is size. Easiest way to make samples is to split the original in HDTV2MPEG2 (set segment size to 9 MB to be safe), run it through mpeg2repair again, and then look at the log to see which was the last segment before the error.
post #89 of 426
Quote:


Originally posted by Wizziwig
Posted new versions of MPEG2Repair (1.0.0.5) and NullPacketSaver (1.02). These updates are to improve user-friendliness/interface. No major changes to functionality. Please keep sending ideas and bug reports for future versions.

Thanks Mark.

I have some AVX1 DirectTV HBO recordings that normally don't run on my Samsung 165, with constant image tear.

Once repaired with the previous version, they seem to do at first, but after a while, the 165 starts to stutter. Hitting Stop/Play makes it work again, but it does not last.

Any idea if this can be corrected? Would you need a sample?
post #90 of 426
Quote:


Originally posted by Wizziwig
Posted new versions of MPEG2Repair (1.0.0.5) and NullPacketSaver (1.02). These updates are to improve user-friendliness/interface. No major changes to functionality. Please keep sending ideas and bug reports for future versions.

-Mark

Mark,

Thanks for the minimize button

Another little bug I've found is when you're processing a file which is on a bad DVD (a dirty DVD for example or scratched or with some burning errors) MPEG2Repair crashes and sometimes crashes the whole PC.

J.
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: HDTV Recorders
AVS › AVS Forum › HDTV › HDTV Recorders › MPEG2Repair: New Tool for Error Detection (also improves 169time compatibility)