or Connect
AVS › AVS Forum › HDTV › HDTV Recorders › MPEG2Repair: New Tool for Error Detection (also improves 169time compatibility)
New Posts  All Forums:Forum Nav:

MPEG2Repair: New Tool for Error Detection (also improves 169time compatibility) - Page 2

post #31 of 426
Quote:


Since you're the mpeg2 master, maybe you can shed some light on the mysterious 0x000001B4 start code. I can't find much information about
it other than "sequence error". Where does this come from? Is it inserted
by the receiver or at the broadcast station? It seems more common for
OTA. Right now, I treat it like any other error but was wondering if
there's something special you can do with it (is it followed by any useful
data?).

The 0x000001b4 start code is the "sequence_error_code". Here's what
MPEG-2 has to say:

"The use of the start codes is defined in the following syntax description
with the exception of the sequence_error_code. The sequence_error_code
has been allocated for use by a media interface where uncorrectable errors
have been detected."

So I'd say that if it appears in the video stream, it's just a warning that
an upstream error correction scheme (most likely a Reed-Solomon
block code error in a receiver) has found an error. You still have to find the
actual bad bits after the sequence_error_code.

Ron
post #32 of 426
Thread Starter 
Quote:


Originally posted by robena
Wiz,

The new JVC 5U does not like streams where commercials have been removed with HDTVtoMPEG2.

Very often, the transition triggers a very noticeable lip-sync delay.

Could your tool be improved to mask the audio discontinuities introduced by the editing, making the 5U usable with these streams?

I'm aware of that issue. It's one of the reasons I stopped using HDTVtoMPEG2 for editing my recordings. I have not used it in a long time but last I checked it also didn't always properly trim the video on GOP boundaries either. I've been using Womble's MPEG Wizard instead. If you enable the 'GOP Trim' option, it produces decent results. But you definitely need to remux the output because it strips all null packets and does some funky things to the data alignment. I also heard it doesn't handle field repeat flags (3:2 pulldown) but that shouldn't matter if you're remuxing.

I wish I could help out on that project but I just don't have the time. Luckily (or not depending how you see it), I was between jobs when I started MPEG2Repair so I had more time to kill. Unfortunately, my program doesn't have the infrastructure necessary to do anything useful with audio data (other than check for continuity errors). To make any audio changes/repair would take a lot of work. I'll keep it on the "to-do" list for the future.

I think the current H2M author is on the right track... so hopefully it will get fixed eventually.

-Mark
post #33 of 426
Quote:


Originally posted by Wizziwig
I've been using Womble's MPEG Wizard instead.

Wizard is just too buggy for that. With some streams it works, and with others it's a real mess.

In particular, it handles drops very badly losing audio sync very easily.

Besides, H2 has a much much faster GUI for editing. With the keyboard shortcuts, it takes around 2 minutes to edit a one hour show.

Quote:


I also heard it doesn't handle field repeat flags (3:2 pulldown) but that shouldn't matter if you're remuxing.

I reported this bug to Womble, and they fixed it, at least for the particular movie I was trying to mux that used the repeat flags.
post #34 of 426
Quote:


Originally posted by Wizziwig
No. As I explained, the program is essentially an MPEG2 decoder. Try finding a software MPEG2 decoder that can decode HDTV on a 1 Ghz machine (like the AVX) without hardware assistance.

It would simply be too slow to do in real-time unless the AVX was upgraded to something like a 2 Ghz+ machine.

-Mark

Many of us have or would get a 2 Ghz+ machine to run a souped up AVX1 with integrated MPEGRepair so that recording/arciving became a i-pass error-masked affair.


Also, it woul greatly improve results when using chasing playback by having MyHD/AcccessDTV playing a recording in progress.
post #35 of 426
Wiz,
I want to thank you for the great software. I tried it out on two 20 Gig per hour HD files that I had recorded from satellite. These became smoothly playable with DVHStools / Ravanescent codex and also with Fusion software. Without repair they would freeze the software at start of play not even displaying a single frame. Elecard player could previously display them but very with very broken audio and video on my 2.24 Gig machine. The repair takes a least twice as long as to play the file but it is worth it.
Thanks John
post #36 of 426
Mark,
Thanks for the detailed explanation of the error log file created by MPEG2Repair. I also started playing with NullPacketSaver. Very nice. Now that we have NullPacketSaver, I would love to see a command line version of MPEG2Repair. This way I can make a batch file which will run MPEG2Repair followed by NullPacketSaver.

The goal would be to simplify the creation of two archives. One archive would be an output file from MPEG2Repair for transfer to D-VHS. The second archive would be null stripped files split into 4.47 GB chunks for DVD-R (along with the .nul file). I've started to do this because it

1. Let's me watch a show just by popping in a D-VHS tape,
2. keeps my hard drives clean, and
3. provides a lossless, error-free backup on DVD-R, and
4. provides a redundant archive, which is good in case one of the archives gets lost or damaged.

I'm sure a command line version of MPEG2Repair would be handy in some other scripted process flows.

-Dylan
post #37 of 426
Quote:


Originally posted by robena

The new JVC 5U does not like streams where commercials have been removed with HDTVtoMPEG2.

Very often, the transition triggers a very noticeable lip-sync delay.


Does HDTVtoMPEG2 cause lip-sync issues with streams archived to a JVC30K if HDTVtoMPEG2 is used to trim the excess material from the beginning and ends of PC HD captures?

What the best program for this simple form of editing?
post #38 of 426
Thread Starter 
Quote:


Originally posted by dahester
Mark,
Thanks for the detailed explanation of the error log file created by MPEG2Repair. I also started playing with NullPacketSaver. Very nice. Now that we have NullPacketSaver, I would love to see a command line version of MPEG2Repair. This way I can make a batch file which will run MPEG2Repair followed by NullPacketSaver.

The goal would be to simplify the creation of two archives. One archive would be an output file from MPEG2Repair for transfer to D-VHS. The second archive would be null stripped files split into 4.47 GB chunks for DVD-R (along with the .nul file). I've started to do this because it

1. Let's me watch a show just by popping in a D-VHS tape,
2. keeps my hard drives clean, and
3. provides a lossless, error-free backup on DVD-R, and
4. provides a redundant archive, which is good in case one of the archives gets lost or damaged.

I'm sure a command line version of MPEG2Repair would be handy in some other scripted process flows.

-Dylan

It would be relatively easy to make a command-line version since that's how it started originally. It also used to process mpeg2 elementary streams but I dropped that support because it didn't seem useful for HD (which usually comes in TS files).

Do you actually need a text-mode version of the tool, or can I simply add command-line arguments for input/output paths to the existing GUI version? The only thing missing from a GUI version would be the error return codes that one normally tests in batch files.
post #39 of 426
Thread Starter 
Quote:


Originally posted by mkerdman
Many of us have or would get a 2 Ghz+ machine to run a souped up AVX1 with integrated MPEGRepair so that recording/arciving became a i-pass error-masked affair.


Also, it woul greatly improve results when using chasing playback by having MyHD/AcccessDTV playing a recording in progress.

Well... I can't really get into the details but patching things after-the-fact like my tool does is really not the appropriate solution. Once data is missing/corrupted there's really no way to restore 100% of the original. If the AVX is still introducing errors (and I'm not saying it is), then it's something that needs to be fixed by Richard/169time. I also feel that my tool should remain an optional process because it is not necessary for all decoders and may not do as good of a job in some situations.

-Mark
post #40 of 426
Quote:


Originally posted by mkerdman
Does HDTVtoMPEG2 cause lip-sync issues with streams archived to a JVC30K if HDTVtoMPEG2 is used to trim the excess material from the beginning and ends of PC HD captures?

What the best program for this simple form of editing?

No, H2 does not. And it's by far the best program to trim TS files.
post #41 of 426
Quote:


Originally posted by robena
No, H2 does not. And it's by far the best program to trim TS files.

Robert

Thanks, that's what I have been using, but, I have not watched all the archives to see if any lip-sync issues existed.

I have never tries Womble which has also been mentioned as suitable for this purpose.
post #42 of 426
Thread Starter 
Version 1.0.0.4 is now available. Very minor/quick changes (see first post). Fixed a crash bug which appeared in one of my recordings and improved default program selection for OTA. It should now pick the first program found instead of the last (usually weather, spanish, etc.).

Have fun.

-Mark
post #43 of 426
Quote:


Originally posted by mkerdman
I have never tries Womble which has also been mentioned as suitable for this purpose.

Womble will give you lip-sync problems galore.
post #44 of 426
Quote:


Originally posted by Wizziwig
Well... I can't really get into the details but patching things after-the-fact like my tool does is really not the appropriate solution. Once data is missing/corrupted there's really no way to restore 100% of the original. If the AVX is still introducing errors (and I'm not saying it is), then it's something that needs to be fixed by Richard/169time. I also feel that my tool should remain an optional process because it is not necessary for all decoders and may not do as good of a job in some situations.

-Mark

Mark

Just last night I watched a 169Time AVX1 D 2-hour feature film DVHS tape capture through an LG 3410 which had a single glitch whose origins are hard to attribute.

From what you have said, had I used the MPEGRepair Tool on a PC capture of the same film and then archived it to DVHS, while improving playback on some decoders such as PCI capture cards software players, it MIGHT HAVE present more glitches on playback through either the JVC 30K or the LG 3410 and possibly the Samsung T165 than the single glitch the direct to DVHS capture did.

If that's true, the MPEGRepair Tool could be made optional upon AVX1 capture if you and Richard were to agree.

My idea before reading your comment above, was to capture everything on PC, repair each capture, and then archive to DVHS.

However, would that possibly result in the 3410 glitching more?

Some testing is required, and, I will do it.
post #45 of 426
Mark,

How can I tell MPEG2Repair that I want the output file to be several files of a specific size ?

Thanks again for this great tool.

J.
post #46 of 426
Thread Starter 
Quote:


Originally posted by mkerdman
Mark

Just last night I watched a 169Time AVX1 D 2-hour feature film DVHS tape capture through an LG 3410 which had a single glitch whose origins are hard to attribute.

From what you have said, had I used the MPEGRepair Tool on a PC capture of the same film and then archived it to DVHS, while improving playback on some decoders such as PCI capture cards software players, it MIGHT HAVE present more glitches on playback through either the JVC 30K or the LG 3410 and possibly the Samsung T165 than the single glitch the direct to DVHS capture did.

If that's true, the MPEGRepair Tool could be made optional upon AVX1 capture if you and Richard were to agree.

My idea before reading your comment above, was to capture everything on PC, repair each capture, and then archive to DVHS.

However, would that possibly result in the 3410 glitching more?

Some testing is required, and, I will do it.

If you don't change any of the default INI file settings, then repair should be very safe. My program will never introduce glitches that were not there to begin with (at least nobody has ever reported such a bug). The reason I would make repair optional is because a good decoder can recover some errors in ways that I can't. Here's an example:

Let's say you have an error in an I-Frame. According to the mpeg2 spec, you're not allowed to skip any macroblocks in such pictures. So when I detect an error on this type of frame, I'm forced to replace any missing/corrupted macroblocks with something valid - I use black macroblocks. But in theory, a decoder could do something else such as skipping the rest of that slice/row and moving on to the next one. If there is no motion in the scene and the decoding buffers contain a previous frame, you may not even notice that those pixels were never updated. But you may notice if black pixels appear there as a result of my repair. Of course for many/most people here their decoder isn't this intelligent and would skip that entire frame, display random garbage, or worse.

My program does its best to regenerate a file that follows the ISO 13818-1, 13818-2 specs. I suggest people look at the log and watch those spots for themselves to determine if they prefer how their decoder handles the error compared to mine. For what it's worth - I watch 169time/Dish6000 recordings on a JVC30K and always run them through my program first. I have yet to find a case where an error looked worse after my program.

-Mark
post #47 of 426
Thread Starter 
Quote:


Originally posted by Johnny Begood
Mark,

How can I tell MPEG2Repair that I want the output file to be several files of a specific size ?

Thanks again for this great tool.

J.

Unfortunately, no. That's why I wrote the extra NullPacketSaver program. Due to the way my program does the repair, it often needs to go over and patch the output file in random places. It would complicate things too much if the output file was split into several segments. It's something I'll look into for the future but it's not likely to be added any time soon.
post #48 of 426
Quote:


Originally posted by Wizziwig
If you don't change any of the default INI file settings, then repair should be very safe. My program will never introduce glitches that were not there to begin with (at least nobody has ever reported such a bug). The reason I would make repair optional is because a good decoder can recover some errors in ways that I can't. Here's an example:

Let's say you have an error in an I-Frame. According to the mpeg2 spec, you're not allowed to skip any macroblocks in such pictures. So when I detect an error on this type of frame, I'm forced to replace any missing/corrupted macroblocks with something valid - I use black macroblocks. But in theory, a decoder could do something else such as skipping the rest of that slice/row and moving on to the next one. If there is no motion in the scene and the decoding buffers contain a previous frame, you may not even notice that those pixels were never updated. But you may notice if black pixels appear there as a result of my repair. Of course for many/most people here their decoder isn't this intelligent and would skip that entire frame, display random garbage, or worse.

My program does its best to regenerate a file that follows the ISO 13818-1, 13818-2 specs. I suggest people look at the log and watch those spots for themselves to determine if they prefer how their decoder handles the error compared to mine. For what it's worth - I watch 169time/Dish6000 recordings on a JVC30K and always run them through my program first. I have yet to find a case where an error looked worse after my program.

-Mark

Mark

I archive primarily to DVHS.

Up until now I have found that the LG 3410 played the old A85 169Time AVX1 tapes glitch-free, and, it does equally well with the 08D pre-release.

The LG 3410 decoder might be correcting errors I never see.

Do you think I would be best served archiving the original capture, or, always using your tool first.

I ask because I may well want to re-archive my DVHS library to Blu-Ray or HD-DVD someday and I have no guarantee how decoders such as those in a next-gen optical recorder/player will handle the transmission/reception/capture errors.
post #49 of 426
still only getting the 03 version off your website...not 04.

Thanks
post #50 of 426
Never mind...it's amazing what a web page refresh will do...
post #51 of 426
Quote:


But in theory, a decoder could do something else such as skipping the
rest of that slice/row and moving on to the next one. If there is no motion in
the scene and the decoding buffers contain a previous frame, you may not even
notice that those pixels were never updated.

No theory required. That's exactly what MPEG-2 decoders do (unless concealment
vectors are present). Here's a graphic example of a single bit error in an
I-frame with low motion:

http://www.avsforum.com/avs-vb/attac...853&fullpage=1

Same I-frame, but with the reconstruction buffer cleared to zeroes before
decoding the frame:

http://www.avsforum.com/avs-vb/attac...859&fullpage=1

Ron
post #52 of 426
Quote:


Let's say you have an error in an I-Frame. According to the mpeg2 spec, you're not allowed to skip any macroblocks in such pictures. So when I detect an error on this type of frame, I'm forced to replace any missing/corrupted macroblocks with something valid - I use black macroblocks. But in theory, a decoder could do something else such as skipping the rest of that slice/row and moving on to the next one. If there is no motion in the scene and the decoding buffers contain a previous frame, you may not even notice that those pixels were never updated. But you may notice if black pixels appear there as a result of my repair. Of course for many/most people here their decoder isn't this intelligent and would skip that entire frame, display random garbage, or worse.

The least noticeable option might be to replace the single bad I-frame block with the corresponding contents of the preceding frame if that's possible. Though I guess that would envolve re-encoding that block.

- Tom
post #53 of 426
Quote:


Originally posted by Wizziwig

Do you actually need a text-mode version of the tool, or can I simply add command-line arguments for input/output paths to the existing GUI version? The only thing missing from a GUI version would be the error return codes that one normally tests in batch files.

I think just adding command line arguments for the I/O files to the existing GUI version would be fine (and of course it would launch into repairing and then exit upon completion without having to press any buttons in the GUI). Thanks again for your efforts.

-Dylan
post #54 of 426
Thread Starter 
Quote:


Originally posted by dr1394
No theory required. That's exactly what MPEG-2 decoders do (unless concealment
vectors are present). Here's a graphic example of a single bit error in an
I-frame with low motion:

http://www.avsforum.com/avs-vb/attac...853&fullpage=1

Same I-frame, but with the reconstruction buffer cleared to zeroes before
decoding the frame:

http://www.avsforum.com/avs-vb/attac...859&fullpage=1

Ron

The reason I said "in theory" is because most of the PC decoders designed for DVD playback don't follow this rule. The Intervideo ones for example, just skip the entire frame. When watching, it shows up as random pauses as it waits to sync up with audio. I guess it depends on the user but I find random black squares less distracting than those pauses. Then there's other errors like bad motion vectors. In theory those should be ignored, but I've seen decoders that simply crash because they don't do any bounds checks on memory buffer access during motion compensation. People really need to check their particular decoder to see what it does.


Quote:


Originally posted by trbarry
The least noticeable option might be to replace the single bad I-frame block with the corresponding contents of the preceding frame if that's possible. Though I guess that would envolve re-encoding that block.

- Tom

Yeah... that's something I'm going to look into for the future. Main problem is that there's no guarantee that the previous p-picture has an intra-coded block in that position and the previous i-picture is likely too far away in time to provide a useful sample. The tool is already doing a re-encode of sorts in order to bring the luminance of the corrupted macroblock down to black. It was the simplest fix I could think of and similar to the DC only intra-blocks originally suggested by Ron. If you check the screen positions of 169time errors (at least on the old software versions) you will see why that strategy usually results in an invisible/perfect repair.


Quote:


Originally posted by mkerdman
Mark

I archive primarily to DVHS.

Up until now I have found that the LG 3410 played the old A85 169Time AVX1 tapes glitch-free, and, it does equally well with the 08D pre-release.

The LG 3410 decoder might be correcting errors I never see.

Do you think I would be best served archiving the original capture, or, always using your tool first.

I ask because I may well want to re-archive my DVHS library to Blu-Ray or HD-DVD someday and I have no guarantee how decoders such as those in a next-gen optical recorder/player will handle the transmission/reception/capture errors.

Well, you can keep doing what you're doing. If the optical devices down the road have problems, you can then run your archives through my tool before storing them to disc. In my experience, most of the hardware decoders should deal with errors. After all, errors are pretty much expected for OTA transmission. This whole issue is more of a problem for the DVD decoders because they expect perfect data to be coming off the disc. While on the archiving topic, I will say that I've had very bad experience with DVHS. I have a refurbished JVC 30K (with all the updates) and have rarely seen a recording without problems. Maybe it's my choice of tape - Fuji SVHS. In any event, I always record directly to hard-drive and archive to DVD-R to avoid introducing additional errors due to tape. I do still use the JVC to watch all my recordings by streaming files to it via firewire. I'm seriously considering getting a MyHD card to replace the JVC.

-Mark
post #55 of 426
I just thought I should congratulate the author on such a great program. Bofore using this software I could not edit my CITYTV-HD recordings of Enterprise, using even the latest beta of HDTV2Mpeg2. After running the files through this utility, voila! , they are fully accesible for editing.

Thanks!


Marc.
post #56 of 426
Quote:


Originally posted by dahester
I think just adding command line arguments for the I/O files to the existing GUI version would be fine (and of course it would launch into repairing and then exit upon completion without having to press any buttons in the GUI).

I second that, I also need to be able to automatize the process.
post #57 of 426
Thread Starter 
Quote:


Originally posted by dahester
I think just adding command line arguments for the I/O files to the existing GUI version would be fine (and of course it would launch into repairing and then exit upon completion without having to press any buttons in the GUI). Thanks again for your efforts.

-Dylan

Quote:


Originally posted by robena
I second that, I also need to be able to automatize the process.

I think you guys probably need a text-mode version. If you're going to launch it from a batch file, how will you know when it's done so you can proceed with the script? The GUI version would return instantly and then continue the repair/scan in the background. You probably need something that doesn't return when executed until it's done. Or are you using a more complex scripting system than simple DOS batch files?
post #58 of 426
Quote:


Originally posted by Wizziwig
I think you guys probably need a text-mode version. If you're going to launch it from a batch file, how will you know when it's done so you can proceed with the script? The GUI version would return instantly and then continue the repair/scan in the background. You probably need something that doesn't return when executed until it's done. Or are you using a more complex scripting system than simple DOS batch files?

I could of course check the size of the file being generated, and decide that the repair is done when it stops growing (I use something more evolved that .BAT files for scripting), but a text mode version will be of course preferable.

The only draw back if that it will make you maintain and post 2 versions instead of one, so that's up to you if you think that's not too much hassle.

Anyway, text mode or returning GUI, THANKS!
post #59 of 426
Mark,

Do you feel that your program is ready to be used with other data instead of just the 169time system?


I am asking because I ran your program on a 27 Gbyte file that was converted from the HDTIVO (Directv High Def receiver/PVR).

it passed with flying colors ie.It found no errors and the repaired file was the same exact size as the original.

HOWEVER, MYHD is obviously having trouble with the file because the video breaks up throughout the file on playback.

Thanks,

Joe
post #60 of 426
May have seen the same thing when processing a file from the R5000-HD. Looking for tools that can generically run on a ts stream whether there is an issue or not.

Unfortunately, I didn't save the original stream so it could have been in the source.
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: HDTV Recorders
AVS › AVS Forum › HDTV › HDTV Recorders › MPEG2Repair: New Tool for Error Detection (also improves 169time compatibility)