How to create 3D frame sequential video? - Page 4

Did you try the new jap.mp4 video?

Mathew Orman

Matthew, I didn't see your post then.
Thanks for this new attempt.
I tried the second file now, both names, as you suggested.

They behave like the first file you sent.

The answer has been given two pages earlier. The guys at doom9 analysed the file and found how sony made it :

There is no commonly available tool that can produce a stream like Sony used.
Unless you are a programmer and you can do a patch for x264 (or make your own tool to convert files).
Adding the SEI messages by hand with a HEX editor should also be possible, but... who could be desperate enough to do it ?

Thank you. It looks like the 3D information is in individual frames. I will debug this and come up with working solution.

Mathew Orman

Try again:

http://icesterf.webd.pl/samples/AVCHD720p.mp4

This one has the Sony PSP profile and the same encoder preset.

Mathew Orman

Thanks,
the performance of this file is different than the others.
On the PS3 is played like VLC plays it on the PC: the PS3 shows 2D and in rapid succession the left and the right eye images.


Another idea (this was suggested to me) to identify the format is the following:
- opening an open-source video-player in a debugging tool such as Visual Studio,
- See how this tool opens the Sony file and decodes/plays the 1st few frames.

In this way, we can mark all those bytes in the Sony file which have nothing to do with the special Sony 3D format (the pure video stream and parts of the container that we know cannot contain the special bytes).
In this way we can focus on relatively small area of the file.

If we do the same with the file generated by Avidemux, by copying the video and audio streams, we should see better what bytes are different. At this point of the analysis, they might be few.

Just an idea.

Yes, very correct in pointing out the state of the art which was described in the previous pages.
Rather than waiting tough, in order to move forward, I would edit the file as you suggested.
If I had the technical skills, to parse the container and the stream and to see what needs to be amended/added and here. I would do it because if the patched video file works, we would have found the algorithm.
If we had the algorithm, even with my limited programming skills, I would be able to write a program to convert files to the PS3 video format.
An inefficient program but it would do.

Mathew is confident that he can create files that are very similar to the Sony one and is confident that he can identify the differences. That process should yield the precise conversion algorithm.


The prize is huge, really: being able to have the PS3 to play any 3D video in stereo natively, rather than relying on very expensive TVs.
The reason why I say the prize is huge is because HDMI 1.4 displays are truly inexpensive now. I was giving the example of my setup where the VIP Gamer + old-school short through HD DLP projector have a cost which is multiples less than a multi-format 3D TV. Apart from big cost savings, there other benefits of playing S3D natively on the PS3 from USB: a display size as big as a wall, you mount it in a minute and, after watching the S3D video, you can play a bit with the Move controller with friends without having to worry that you interrupt the flow of light between the projector and the wall.
The PS3 community would be very happy.

Apart from all the currently available content in S3D, yesterday I went to an official Sony shop and asked the official expert there how can the Sony user will store and play the 3D videos made by Sony HDRTD10 new 3D camcorder.
He said categorically that the PS3 would not play such videos from the USB disk and the user would have to create a 3D Blu-ray.
But of course, the PS3 has got the ability to play 3D videos natively from the USB disk. Converting to this MP4 stereo 3D Sony format is a problem begging to be solved.

Sony Vegas has proven useless as it does not even have Sony PSP profile.
Instead I have installed professional AVC encoder made by MainConcept which allowed me to create almost identical preset. I will now need better h264 header viewer to verify that there is in deed a dedicated flag that indexes lef and right images. I may have to purchase PS3 to speed this process as I also have an interest in producing PS3 compatible 3D content.

Mathew Orman

I think there is no need for adding eye sync info in form of new (Sony proprietary) structures since it can be done by using appropriate key frame distance in encoding schema.

The problem at the moment is the Mediainfo application which reads the mp4 structure and reports format setting of GOP in M:x N:x format which is not sufficient for debugging B frames. So I need similar program which reports GOP in I:x P:x B:x format. With such, the encoder presets would be complete and only then I would know if there are any Sony custom eye order indexing.

I see someone is already requesting such feature for Mediainfo application:

http://sourceforge.net/tracker/?func...62&atid=581184


Mathew Orman

This is what Mediainfo reports on original Sony 3D content and for what I have tried to make a match for:
------------------------------------------------------
General
Complete name : C:\\Users\\User\\Desktop\\Rapid upload\\E320103DSizzle_Trailer-3D720.MP4
Format : MPEG-4
Format profile : Sony PSP
Codec ID : MSNV
File size : 200 MiB
Duration : 2mn 19s
Overall bit rate : 12.0 Mbps
Encoded date : UTC 2010-08-04 15:57:27
Tagged date : UTC 2010-08-04 15:57:27

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.0
Format settings, CABAC : Yes
Format settings, ReFrames : 4 frames
Format settings, GOP : M=1, N=56
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 2mn 19s
Bit rate mode : Variable
Bit rate : 11.8 Mbps
Width : 1 280 pixels
Height : 720 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 59.940 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.213
Stream size : 196 MiB (98%)
Language : Japanese
Encoded date : UTC 2010-08-04 15:57:27
Tagged date : UTC 2010-08-04 15:57:27
Color primaries : BT.709-5, BT.1361, IEC 61966-2-4, SMPTE RP177
Transfer characteristics : BT.709-5, BT.1361
Matrix coefficients : BT.709-5, BT.1361, IEC 61966-2-4 709, SMPTE RP177

Audio
ID : 2
Format : AAC
Format/Info : Advanced Audio Codec
Format profile : LC
Codec ID : 40
Duration : 2mn 19s
Bit rate mode : Variable
Bit rate : 256 Kbps
Maximum bit rate : 384 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Compression mode : Lossy
Stream size : 4.27 MiB (2%)
Language : Japanese
Encoded date : UTC 2010-08-04 15:57:27
Tagged date : UTC 2010-08-04 15:57:27

------------------------------------------------------------
General
CompleteName : C:\\Users\\User\\Documents\\Adobe\\Premiere Pro\\5.0\\AVCHD720p_1.mp4
Format : MPEG-4
Format_Profile : Sony PSP
CodecID : MSNV
FileSize/String : 25.5 MiB
Duration/String : 26s 218ms
OverallBitRate/String : 8 166 Kbps
Encoded_Date : UTC 2011-03-11 08:27:19
Tagged_Date : UTC 2011-03-11 08:27:19

Video
ID/String : 1
Format : AVC
Format/Info : Advanced Video Codec
Format_Profile : High@L4.0
Format_Settings_CABAC/String : Yes
Format_Settings_RefFrames/String : 4 frames
Format_Settings_GOP : M=1, N=56
CodecID : avc1
CodecID/Info : Advanced Video Coding
Duration/String : 26s 176ms
BitRate_Mode/String : Variable
BitRate/String : 8 018 Kbps
Width/String : 1 280 pixels
Height/String : 720 pixels
DisplayAspectRatio/String : 16:9
FrameRate_Mode/String : Variable
FrameRate/String : 59.940 fps
FrameRate_Minimum/String : 39.973 fps
FrameRate_Maximum/String : 59.940 fps
Standard : NTSC
ColorSpace : YUV
ChromaSubsampling : 4:2:0
BitDepth/String : 8 bits
ScanType/String : Progressive
Bits-(Pixel*Frame) : 0.145
StreamSize/String : 25.0 MiB (98%)
Language/String : English
Encoded_Date : UTC 2011-03-11 08:27:19
Tagged_Date : UTC 2011-03-11 08:27:19
Color primaries : BT.709-5, BT.1361, IEC 61966-2-4, SMPTE RP177
Transfer characteristics : BT.709-5, BT.1361
Matrix coefficients : BT.709-5, BT.1361, IEC 61966-2-4 709, SMPTE RP177

Audio
ID/String : 2
Format : AAC
Format/Info : Advanced Audio Codec
Format_Profile : LC
CodecID : 40
Duration/String : 26s 218ms
BitRate_Mode/String : Constant
BitRate/String : 157 Kbps
Channel(s)/String : 2 channels
ChannelPositions : Front: L R
SamplingRate/String : 48.0 KHz
Compression_Mode/String : Lossy
StreamSize/String : 504 KiB (2%)
Language/String : English
Encoded_Date : UTC 2011-03-11 08:27:19
Tagged_Date : UTC 2011-03-11 08:27:19

-------------------------------------------------------------

Mathew Orman

Very good progress Mathew.
It seems to me that, concerning the information that MedioInfo can capture, the two video are the same.
Which of the videos is the one you compared? One of those you gave us to test or a new one?

So, from your post before the last one, I understand that you are now moving towards debugging the extra info in each frame.
Please keep us updated.
Do we know what tool PigPines used for his analysis, the analysis that BlackShark pointed our attention to.
Also, was that analysis complete? I have the impression that BigPines implemented the changes he identified but it still did not work on the PS3.

I have just ordered PS3 to speed the debugging and test the performance of the PS3 Game SDK.

The two headers from my previous post are: first one is the original video that works and second is the same I have posted a link to.

Should have more information on this mysterious Sony 3D frame sequential format soon.

Mathew Orman

Great Job on this guys.

Did any of the files work if you set the tv/projector to frame sequential.

Mathew,
it is great that soon you will receive the PS3 !
(we were seeing the Sony MP4 in native HDMI 1.4 on firmware version 3.55)

Also, a very good idea to look into its SDK: maybe it is explained there how to create this type of video files: they might be useful to PS3 developers to show in-game 3D footage.

Please keep us updated

At this point, I am just guessing like everybody else. The guys on Doom 9 helped me get as far as I did.

I am a software developer but I simply don't have the experience to fix this problem on my own. Since I couldn't seem to get anyone interested in patching x264, I attempted to patch it myself without success.

I am still very interested in this project and would love to help any way I can.

Mike

Mike,
I am happy to hear that you are still very interested in this project.

You did very nice progress on the project.
Mathew is progressing nicely.

Is there is a way for you to get more information about the Sony format from the Doom 9 guys? I understand that they have no interest in changing the codec (for lack of PS3 or 3D display) but it would still be very useful to nail down the PS3 format in detail.
My impression is that Mathew is now alone in this very difficult objective and I hope that you can collaborate because to solve these complex issues alone is particularly challenging.

If we had the PS3 format nailed down, apart from updating the coded, we could manually create a file with a dozen frames and make sure it works.
Also we could create a program to patch files similar to the PS3 format to make them 100% PS3 compliant.
At that point, it should be easier to get the codec upgraded.

For anyone interested, the Doom 9 thread discussing these issues is:

http://forum.doom9.org/showthread.php?t=158457&page=17

Mike

How frame packing relates to x264 compression is a mystery to me.
Could anyone explain?

Mathew Orman

OK
They used the name frame packing which was already defined in HDMI 1.4a video transmission protocol.
The correct name for it is frame compatible stereo.
Such has nothing to do with Sony's frame sequential stereo
so the SEI messages are irrelevant and for sure Sony did not use it as it only applies to frame compatible stereo.

Mathew Orman

Frame packing is a generic term that was already used previously, for example it was used for naming one of the hacks (B-frame packing) that allows storing MPEG video (MPEG1, 2 or 4) into .AVI containers.

The official name used by MPEG for this SEI message is "Frame Packing Arrangement SEI Message" and has been added to the MPEG4 AVC standard.
Sony uses it and x264 writes it. The problem is that they are at the moment the only implementations of this SEI message and they write it differently.

Hdmi doesn't hold a trade mark on "Frame Packing".

I do not think so.
Sony uses MainConcept and they did not implement such method of frame indexing yet.

In any case I will nail this problem soon.

Mathew Orman

The user at doom9 said the SEI messages were present for every frame in Sony's file.
Is this correct or incorrect ?

He did not answer when ask how he arrived at such conclusion or what tools he used to debug the transport stream in the Sony's 3d movie sample.


Mathew Orman

Here is the answer he gives at Doom9 forums

h264_parse does not work with mp4 files.
And hex dump will not index
set current_frame_is_frame0_flag.

So, it does not make sense at all.

Mathew Orman

Extract the data by hand from the mp4 file using a hex editor into a raw elementary stream, then open the raw stream with h264_parse.
That is what I understand.

It looks scary. Just out of curiosity, how garbled does mp4 files look ? (no I don't want to know, I don't want to have nightmares, I shouldn't ask that or I'll have a look and be doomed)

And how would do that by hand?

Mathew Orman

any update on this. I would be very interested since I would like to but a camcorder and not use side by side.

Thanks

I have got no news on how to create the MP4 video that the PS3 can play natively in 3D.
I am referring to this:
http://poseidon.dl.playstation.net/c...084&country=us


I installed the latest Sony Vegas Pro 10 (version 10d).
It does not recognise the Sony Mp4 as 3D and it does not have "frame sequential" among its many 3D outputs.

The only partial news, at least for me, is that Vegas Pro says that the AVC encoder used to create the Sony file is called:
HMMP Video Encoder 1.10
(the latest version seems to be 1.14 and the developer seems to be Sony).

By searching the net, I could not identify how to get this encoder nor whether it is part of a professional 3D authoring sw.

As an additional attempt, I hoped that the 3D video encoder was in the latest PS3 SDK 1.92 that leaked publicly but I could not find it there.

Any news on this yet