or Connect
AVS › AVS Forum › Video Components › Home Theater Computers › HDTVtoMPEG2 latest version
New Posts  All Forums:Forum Nav:

HDTVtoMPEG2 latest version

post #1 of 2239
Thread Starter 
I've posted the latest version of HDTVtoMPEG2:

http://www-personal.umich.edu/~balazer/HDTVtoMPEG2/

I've re-united the readme and attempted to update the change log. This version includes the fixes in 1.10b and adds additional fixes by Cris Moore.

I can continue to host HDTVtoMPEG2 here for the time being, so if there isn't a more appropriate person or place to host it, please send your bug fixes to me.


Some basic info about HDTVtoMPEG2:

HDTVtoMPEG2 is primarily an MPEG-2 transport stream editor. Editing is performed as a simple transport packet level cut & splice, which means the output has a transport stream discontinuity at the splice points. Different decoders handle discontinuities with different amounts of grace. You may experience audio and/or video break-ups at the discontinuities. If you need editing with no discontinuities in the output, check out VideoReDo.

HDTVtoMPEG2 can break the output into multiple file segments. It can strip unwanted programs and PIDs from the input, and it regenerates tables. It can optionally preserve the transport stream bit rate (to keep a constant bit rate stream as such).

HDTVtoMPEG2 can scan for commercials by detecting black pillarbox bars.

I think the transport stream to program stream conversion was broken with newer versions of HDTVtoMPEG2. There are probably better programs to use for that. (X-Muxer or VideoReDo?)
post #2 of 2239
HDTVtoMPEG2 (often referred to as "H2" or "H2M")
Current version: v1.11.94
The current version is available here:
http://www.eecs.umich.edu/~balazer/HDTVtoMPEG2/

What's it for?
Despite its name, HDTVtoMPEG2 main purpose is to edit commercials out of HDTV transport streams and write out another transport stream. It still has the capbability to convert to program streams (mpeg2) but this area has received little to no work since v1.10b. If you need this capability you should use another tool like VideoRedo.

HDTVtoMPEG2 Feature\\Bug fix history
Feature\\Bug fix history (v1.11.nn)
.94 - Fixed improper Output file name and crash when deleting from Input box introduced with tooltip change.
.93 - Input list box now only shows the file name. Tooltip shows the path.
.92 - Recognize .rec and .mpg as valid transport stream extensions
.92 - Fixed a bug that could cause the dreaded "Could not find a Channel" bug on valid TS files.
.91 - Clips button state is now saved in the ini file
.91 - Added grey color bar support during commercial scan.
.90 - Relaxed the color requirement for pure black on 4x3 color bars.
.90 - Fixed "The parameter is incorrect" dialog message when you scroll the mouse around in an empty Process list box.
.90 - Fixed code so it could handle TS streams with section encapsulated data, i.e. pointer_field != 0.
.89 - Fixed crash caused when no channel is found
.89 - Fixed crash during loading of some TS file when calculating avg TS rate
.89 - Updated to Visual Studio 2005
.88 - Fixed Max Size dialog popup bug when entering a size larger than 99999
.88 - Fixed invalid file name problem related to very large filenames when using Add
.88 - Fixed crash when using Preserve Bit Rate on some transport streams
.88 - Fixed bug that incorrectly extracted the PCR during PMT validation
.87 - Removed the transport stream extension case sensitivity.
.87 - Option to change the vertical_size_value in the sequence_header from 1088 to 1080
.87 - Added a Configuration menu
.87 - New Cut region marker. Forces a new file to be created at a specified point
.87 - Ability to re-order the input file list by dragging items with mouse
.86 - Region markers are now saved when closing H2M.
.86 - Fixed bug where shortcut keys would not work immediately after Deleting a file from the process list box
.86 - MaxSize, VideoPID, and AudioPID boxes now handled properly when shortcut keys are being used
.85 - Added Alt+D\\d to delete region marker
.85 - Default Channel is now auto checked
.85 - Added an RUSure dialog when the Clear All region markers button is clicked
.85 - Fixed bug where a .TRI file was not created if a .TRI file was selected as the input file
.85 - Region info is no longer saved when Clear All is selected, only when Process\\Include\\Exclude\\Delete\\Move
.84 - Fixed missing TS info (Resolution, FPS, etc.) for some streams
.83 - Fixed possible crash when editing low bitrate SD channels
.83 - Added message box indicating SD channels are not supported with Commercial Scan
.82 - Attempt to include black frames before and after program. Disable with IncludeBlackFrames=0
.81 - UI indicates Avg. TS Rate now instead of bogus Video rate
.81 - Weighted average TS Rate is now calculated instead of using a fixed 19.2Mbps rate
.81 - Option to select 10 second Commercial Scan rate instead of the default 60 seconds
.80 - Re-work of Commercial Scan code to fix troublesome ABC\\FOX stations
.79 - Small tweak to code that determines frame type
.78 - Fix bug that could cause the second file to be ignored during scan if Exclude section had been specified
.77 - Fix bug that would sometimes not let you type letters in the Output FileName and Ext boxes
.76 - Fix to scan code to handle pre-defined Include\\Exclude sections correctly during scanning
.75 - F5 refreshes the input file data if you are editing a TS file that is currently being recorded.
.74 - Files are now opened with FILE_SHARE_READ and FILE_SHARE_WRITE
.73 - Minor changes
.72 - Ability to disable .TRI file creation using the CreateTRIFiles=0 flag in the ini file
.72 - .TRI files not created until an Include, Exclude, button is pressed
.71 - Fixed major bug when Scan filters were being used
.70 - New Commercial Scan code (HDTV support only)
.70 - Jockeyed around the manual editing buttons
.56 - Changed default Channel to now be the first stream with a horizontal size >= 1280.
.55 - Default output file type is now transport stream instead of mpeg2. Last used setting is still saved in the .ini file.
.55 - File type list order is now Transport Stream, HiDTV, Mpeg2.
.55 - NOTE: You will need to manually update the Ext edit box
.54 - Default Channel is now the one with the lowest video PID number
.53 - Fixed possible crash caused by code being too slack when parsing for video start_codes.
.52 - Fixed bug where channel would not be displayed if the audio was of stream_type 0x06(PES private data).
.51 - Commented code out that ignored PMT's with CA_descriptors(build 49). This didn't always indicate that the channel was scrambled.
.50 - Fixed possible crash do to too aggressive of checking for corrupt PMT
.49 - Ignore PSIP packets for Channel info if any of the PMT's contain CA_descriptors
.48 - Fixed bug where audio was not being mapped properly if the PMT was incomplete.
.47 - Fixed possible crash when editing QAM files.
.46 - Fixed. If the KeepPSIP and KeepNULL keys were missing from the ini file(you were using an older Beta), H2M would keep the PSIP and NULL packets until you had de-selected them.
.45 - Fixed bug if "PreserveBitRate" is missing from the ini file H2M will ALWAYS include NULL packets
.45 - Fixed. When using "PreserveBitRate" a small number of packets would get stripped instead of being converted to NULL packets
.44 - Fixed PID swap bug when 2 PMT's referenced the same video PID
.43 - Fixed bug introduced in build 39. Files would not load if duplicate file was encountered
.42 - Fixed PID drop down list box for W2K displaying only 1 entry
.41 - "PreserveBitRate" option flag available in ini file
.40 - Increased scan buffer size a bit
.39 - 0 byte transport streams are now discarded with warning.
.38 - commented out some new code that could cause a crash if the MGT had a large number of tables
.38 - PID swap bug fixed
.37 - Fix to bug introduced by code to detect corrupted PAT and PMT
.36 - New Clip feature
.35 - .TRI files not created now until the "Include" or "Exclude" button is pressed
.34 - Contrast and Brightness settings saved
.33 - detect and strip corrupted PAT and PMT
.32 - write multiple PAT and PMT's at the beginning of the file to make sure H2M will have Channel info to display
.31 - PID 0x1fef recognized as a NULL packet (old HBO recordings?)
.31 - Fixed small memory leak
.30 - ??

1.11 Beta
NEW
- PID stripping support
- MPEG-2 audio now supported when converting TS to MPEG-2
- Option to create sequential "clip" files
- Ability to preserve the stream bit rate using the "PreserveBitRate" flag in the ini file.

FIXED
- Fixed an old memory leak in scan_channel_info() function
- Fixed problem with some transport streams not displaying a preview when "Show Progress Image" is checked
- Fixed missing preview display on Progress dialog for MPEG-2 conversion on some transport streams
- Fixed missing video\\audio on MPEG-2 conversion for some transport streams
- Fixed input path not being applied after selecting files from Explorer(Note: if a .tri file is selected, the path is set to the path to the transport streams NOT to where the .tri file was selected)

1.10.6
NEW:
- Drag and Drop support for TS and .TRI files onto H2M desktop icon from Explorer
- Double click of .TRI file in Explorer starts H2M and auto loads files

FIXED:
- Tweaks to scan_channel_info() to improve pid detection

1.10.5(Merger of Alan Haverson's code and Cris Moore's code)
NEW:
- Hot key short cuts
- Basic Contrast\\Brightness controls
- Specify fixed location for .TRI files through .INI file

FIXED:
- Possible freeze during file transitions

1.10.4
NEW:
- Auto save of region edits
- Mouse wheel steps by i-frame

FIXED:
- >>| Seek to End would not move the bar to the end of the slider
- Crash in Progress dialog when showing progress image
- |<< Seek to beginning would not display first i-frame
- Channel resolutions were sometimes swapped

1.10.3
FIXED:
- Resolution\\FPS\\Aspect Ratio\\Bit Rate not displaying for some transport streams
- No preview picture for some 720p streams
- No preview picture for WB HD transport streams recored with TSReader
- Channel display now favors PSIP data if available and now displays digital and analog channels. If no PSIP data is available, you will get the familiar NONAME channel

History of HDTVtoMPEG2
Year 2002 - Ben Cooley
v1.00 to v1.07
The original developer of HDTVtoMPEG2. HDTVtoMPEG2's original purpose was to extract program streams from transport streams and create a mpeg compatible file that was playable by media players and compatible with video converter tools like VirtualDub. Hence the name of the tool, HDTVtoMPEG2. This original version of HDTVtoMPEG2 did not have any transport stream editing capabilities, it was strickly a conversion tool.

Ben Cooley's original version of HDTVtoMPEG2 v1.07 is currently available here.
http://www.midwinter.com/~bcooley/

Year 2002 - Grant Reeve
v1.07 to v1.10b
The second developer to work on HDTVtoMPEG2. Grant Reeve's effort into H2M went into adding HDTV editing capabilties. His version allowed you to view and edit the stream on I-Frame bounderies and write out a new transport stream minus the "cut out" parts.

Grant Reeve's original version of HDTVtoMPEG2 v1.10b is currently available here.
http://www.derangedkiwi.net/index.php?view=hdtvtompeg2

Year 2003 - Alan Halverson, Cris Moore
v1.10.3 to v1.10.5
These versions continued with the focus on improving the HDTV editing capabilties and robustness of the tool. New features like "hot" keys, the ability to use the mouse to step from one I-Frame to the next, contrast and brightness, and the ability to save edit points were added in these versions.

Year 2004 - 2007 - Cris Moore
v1.10.6 to v1.11.94
These versions continued with the focus on improving the HDTV editing capabilites and robustness of the tool. New features like PID stripping, Clips output, and new Commercial Scan were added in these versions.
post #3 of 2239
Thread Starter 
Great, just let me know if you're going to give it a new home.
post #4 of 2239
I'd like to pass on a big THANK YOU to Cris for the time he spent improving HDTVtoMPEG2.
post #5 of 2239
Thanks for the great work!

Can I ask for one small thing.
Can you make the mouse wheel act like the left and right arrows (forward and reverse 1 time slice)

That would make editing even easier.
Thanks!
post #6 of 2239
One thing I've noticed is that HDTVtoMpeg2 is having troubles reading the transport streams capped from the new cable boxes with firewire.

Any ideas as to why?

Some representative samples are at :

My Website

All of the above sometimes load with incorrect resolution readings.

All of the above always have incorrect time length readings.

And all of the above WILL crash hdtvtompeg2 if you try to scroll within the file. Editting also is choppy (it's like the timeslices are way too far apart).

thanks !

- litz
post #7 of 2239
Is there a file size limit in this version?

I've tried to edit two HDTV broadcast movies and the application always crashes around 9.5 GB when converting.
post #8 of 2239
litz,
- Problems with TS files
I'm downloading your sample files and will look at then when I get a chance. Please be patient though, debugging display preview problems takes a lot of time and effort.

- Timings are wrong
I believe H2 uses the bit rate specified in the TS file for the Video Rate and for calculating the times. Tools that remove items like NULL packets and PID's change the TS file from a CBR to a VBR file and the times are no longer accurate. I have the same problem when I record with TSReader when I have it remove NULL packets and PID's. I believe STB are removing items like NULL and possibly PID's(sub channels) too in order to save HD space. To fix this problem, I believe H2 would probably have to be changed to calculate the average bit rate of the TS file itself in order to be more accurate on the times.

Cliff,
This is probably a side effect of a "fix". Is the file your using 1 big huge file?
post #9 of 2239
"Cliff,
This is probably a side effect of a "fix". Is the file your using 1 big huge file?"


Both

One was 17+ GB recording from ABC (720p) and the other was 1080i (NBC) recording with 32 files of 650 MB. Both crashed around 9.5 GB processed file size.
post #10 of 2239
Quote:


Originally posted by Cliff Watson
"Cliff,
This is probably a side effect of a "fix". Is the file your using 1 big huge file?"

Both

One was 17+ GB recording from ABC (720p) and the other was 1080i (NBC) recording with 32 files of 650 MB. Both crashed around 9.5 GB processed file size.

Cris,

The NBC 1080i file were actually 38 files (650 MB each) so I split them into 3 groups of 13 and the application didn't crash.
post #11 of 2239
Cris - no problem on the wait, these things do take time.

That is representative of the three different types of streams you can capture off a Motorola cable box (via the Mac firewire method) and they don't work very well with H2, so there's something fundamentally different about them versus a traditional MyHD .ts capture.

MyHD has fits playing them too, but it eventually gets around to it.

The goal is to figure out a way to transcribe these to something more "normal" so not only tools but playback works better.

- litz
post #12 of 2239
Oh ...

In addition to the other clips, there's one more :

crash.m2t.

This is *guranteed* to crash hdvtompeg2 when you try and scroll within it.

Load load, and click the forward 1 arror a few times, bang!

Same dir as before on my web site

thanks !

- litz
post #13 of 2239
the problems you're having with .ts files from your Motorola box do NOT occur with my Scientific Atlanta box....

post #14 of 2239
litz,

"All of the above sometimes load with incorrect resolution readings."

I wasn't able to reproduce this with the files you supplied. What other resolutions are displayed?

"All of the above always have incorrect time length readings."

Covered in an above post.

"And all of the above WILL crash hdtvtompeg2 if you try to scroll within the file. Editing also is choppy (it's like the timeslices are way too far apart)."

Wasn't able to make H2 crash with the files you supplied by scrolling withing the file. Perhaps the sample files are too small. I wasn't able to download crash.m2t. The website is no longer available.

Choppy editing - I counted the i-frames in 3 of the samples you supplied and H2 displays all of the i-frames. Granted some of the preview pics don't look so good(528x480i) I think because of the scaling, but they are displayed. Again, perhaps the sample files are too small and I need a bigger file to see the problem.

Cris
post #15 of 2239
I don't know what's going on with the domain.

For some reason, an old IP is coming up in a lot of nameservers.

It should be 24.30.63.124 so if you're not getting that as the IP, your DNS is broken ...

If still no go, please post back and I'll provide a FTP link.

As for the crashing, I cannot scroll at all in ANY .m2t file I have. Two or three clicks, BAM.

They also will not write out to .mpg, but they WILL write out to .ts

Is there any way to *verify* the version of HDTVtoMPEG2 I'm running?

Yes, I've redownloaded it, but that doesn't mean windows doesn't have some cached copy sitting around it may be running instead ...

Is that something that can be easily added to the current version?

thanks

- litz
post #16 of 2239
Oh ...

Also - I've verified these crashes on multiple computers each with different OS (2k / xp) and processors (athlon vs p4) ...

Two or three right arrow clicks, bam.

thanks !

- litz
post #17 of 2239
okay,

Unless and until this DNS issue gets resolved, directories are NOT going to work on my web site. For some reason, an old IP has been distributed internet-wide for DNS for this connection.

Until the fix propagates, it's dead.

If anyone has any ideas on alternates, please let me know ...

thanks

- litz
post #18 of 2239
Nevermind, it's fixed now (gotta love tech support).

As before, sample files are at :

http://terminus.litz.org/hdtvtompeg2

thanks !

- litz
post #19 of 2239
litz,

I'm downloading crash.m2t right now. Its going to take a hour or so.

choppiness -
I just wanted to make sure you understand that H2 only shows the i-frame pictures in a stream. The p-frames and b-frames between each i-frame are not decoded since H2 only allows editing on i-frame boundaries. The number of i-frames available in a stream will determine how smooth the "steps" are between each click. The number of i-frames available in the stream is up to then encoder I would guess.
post #20 of 2239
By "choopiness" I mean two things ...

1) large jumps from one Iframe to another. If I decode to mpeg2 video and frame step it in, say, virtualdub-mpeg2, I can see it's missing I-frames as it steps

2) occasionally there is NO frame displayed at all for a step, or it's the same as the previous step.

- litz
post #21 of 2239
litz,

I wasn't able to get crash.mt2 to crash while clicking or dragging the slider back and forth. I don't know what is causing the crashes on your systems. I've tried downloading the exe on balazer's web site and tried it on a Win2K system. Still no crash.

As far as version, on the left side of the title bar, if you click on the little icon, and then select About HDTVtoMPEG2, the version should be 1.10.3.

choppiness -
Here are the number of i-frames that are in some of the file samples you sent me:
1920X1080i - 63 i-frames
720X480i - 30 i-frames
528X480i - 53 i-frames
I verified the number of i-frames using Mpeg Stream Eye v1.02. When I load 1920X1080i into H2, I can click the right arrow 63 times and walk through all of the i-frames. Each one is different except for the first one and there is never any blank preview. I don't know whats going on except that you may not be running 1.10.3 but an older 1.10b version?

Cris
post #22 of 2239
Just checked - version reported IS 1.10.3

I cannot scroll through these .m2t files, period.

Bang, crash, very unstable.

This is on multiple OS's, multiple computers, different processors.

Got any ideas?

If I had some way to give you a crash report other than windows generic "the program stopped working", that would be good ...

What are you compiling with?

The "crash.m2t" can be made to crash very easily, here ...

Just load it into hdtvtompeg2, then click on the scroll bar.

Now right arrow repeatedly until it crashes. Won't take more than 4 or 5.

I also get massive numbers of blank preview window when single step scrolling through the file.

I could put up with the occasional crashes (I can get around them, once I learn where they occur) if I could actually SEE the frames I was editting.

But since so many are blank, I can't.

A couple of questions, unrelated though ...

1) what would it take to be able to save/load the edit points?

and

2) the >>| symbol does not take the scrollbar pointer to the endpoint of the file

thanks !

- litz
post #23 of 2239
Hmm

Ok, I went through and zapped EVERY copy of hdtvtompeg2 I had on any system and replaced with your binary specifically.

I did find multiple instances of the program on several machines.

It seems to work now.

The version I had *WAS* 1.10.3, but maybe not your compile?

It was shipped with Visual C++ 7.0 dlls, and yours is not.

Dunno ...

The ability to save/load the edit points and fixing the >>| button would still be nice though ...

Thanks for checking all this, however ...

:-)

- litz
post #24 of 2239
One of the very first versions of 1.10.3 that I sent to someone did include separate dll's. Since then, I been compiling using MFC as a static library so I don't have to include the dll's separately.

Glad you got the problem figured out. I was running out of ideas.

Quote:


The ability to save/load the edit points

Funny you should mention that. I've just about got that finished.

The way it works is that when you select "Convert" it will automatically create a "tsri.tri" file in the same directory as the TS file. tsri stands for "TS Region Information". If H2 crashes, restart H2, select Add, and then under "Files of type" select the new option "TSRI Files", then select the tsri.tri file. This will automatically load all of the TS files that were loaded before and reset all of the edit points to the way there were.


Quote:


fixing the >>| button would still be nice though ...

Yes I've noticed that too. Will ad it to my list of things to look at.

Also, I noticed with your file sets that none of them will display a preview in the Progress dialog box when you select the "Show Progress Image". Yet another problem to look at.

Cris
post #25 of 2239
I spoke too soon.

I can't convert any of these .m2t source streams to .mpg.

All I get are 3kb sized .mpg files.

Any ideas?

Also - on the edit points, it would be nice to be able to save the file without starting the conversion, that way you have the ability "recover" if there's a crash while editting.

thanks

- litz
post #26 of 2239
I haven't done ANY work on the problems with converting to .mpg. I've never used the option so I don't even know what the problems are.( Well, I guess I know one now ) When I've gotten some more of the TS editing problems solved I'll take a look at the mpg conversion problems. Right now thats kinda low priority for me.

Quote:


it would be nice to be able to save the file without starting the conversion

Being able to save at any time sounds like a good idea. I'll add yet another button to the dialog.

Cris
post #27 of 2239
Okay, sounds cool.

Meanwhile. v1.09 does write .mpgs, but it skips the very first I-frame (it starts writing the .mpg at the 2nd I-frame so you lose the first few bits).

also, if you want some eye candy I put some other stuff for you to play with in the hdtvtompeg2 directory at the previously mentioned URL ...

- litz
post #28 of 2239
litz,

The ability to save\\load your edits is done.

Just need to do a little more testing with it.

Cris
post #29 of 2239
Hi Cris,

Very rarely, H2 can crash before you click on Convert.

It would be nice if the TSRI file was updated every time you add or modify a region boundary.
post #30 of 2239
Hi Robert,

That should be easy to add. If I do that, does that mean we could get away without having a "Save Edits" button on the dialog? It would just be automatic then right?

Cris
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Home Theater Computers
AVS › AVS Forum › Video Components › Home Theater Computers › HDTVtoMPEG2 latest version