+This is about rifting all sources. The part about how to enable 3D sterescopic Playstation and Xbox is removed because of legal HDCP issues.+
at first I should mention that I am working blind on this project - I mean without a rift. So there is no certainty for me at all if my achievements are of any value. But I have a good feeling, because at least based on my fundamental ideas, better capture software can be made to shorten the gap between real Oculus Rift content and realtime rifted content.
I use video processors to alter video signals - scaling, framerate conversions/double and triple flashing, colour space conversions, 3D format conversion, deinterlacing, colour/brightness/contrast/gamma correction, aspect ratio control/zoom/cropping, softedge blending, geometry corrections via warp map and - very important - motion interpolation. But with every new format or industry standard, a stand alone video processors becomes outdated. And they are limited to what they were intended for, so only a few functions from above per a very expensive device.
So my idea was to set up a video processor for the Rift, that is based on a low level PC (equipped with a Blackmagic Intensity Pro Card), and is capable of applying the following (in real time) to any external source. A computer that can do that to a source is for simplicity named Rift PC in the further text:
- the geometric correction needed by the rift
- scaling to the resolution/refreshrate accepted by the Rift
- convert 2D/3D formats to the side by side format of the Rift
- colour/brightness/contrast/gamma correction
- aspect ratio control/zoom/cropping
- motion interpolation (the software I use so far - Stereoscopic Player - can easily integrate the "Smooth Video Project" expansion and apply motion interpolation to video files. Until now, I was not able to achieve this with external inputs. But future software solutions, or updates to recent ones should enable motion interpolation for external sources. But not at the cost of more latency - so effective interpolation modes are needed.
Thanks for reading so far. For your patients, and to motivate you to continue reading, I add a small video here. It shows the low latency with a Xbox 360 being rifted - on the fly/live/realtime.2D HDMI XBOX works directly with HDMI ports of a Blackmagic cards, as a 2d XBOX signal is not HDCP protected:
XBOX 360 latency test while realtime rifted for Oculus Rift (e5200 WinXP):
Edit: New video XBOX 360 latency test while realtime rifted for Oculus Rift (Phenom X6 Win7). Read description in post 8 below - no detectable latency:
There is small latency, but please consider what is done to the 720p60 output of the 360: Scaling to 1280x800, geometry correction, and (if needed) 2d(3d) format conversion to side by side. More important is that the software (Stereoscopic Player) was never optimized for low latency, nor is a standard Win XP computer I used as Rift PC. I run a not loaded XP system in conjunction with an E5200, 2gb 800mhz RAM and a Nvidia 7200. Why so lame? Because a Windows 7 computer based on a recent Intel Quad at 4ghz, 2400mhz Trident RAM and a 7-Series GTX (thanks Matthias for your water-cooled beast), because this computer ended up with a huge delay on the real time playback. Both Rift computers had no firewall installed, nor where loaded with updates or software. But my XP based capture PC, after disabling GPU scaling and a few other things, was by far faster. So a "2400mhz RAM" Rift computer, with a optimized/lean operating system and an optimized software solution should give an even better latency.
Talking about latency: It can be considered general knowledge, that a "cable" controller gives a better latency in games than a cordless controller. So I wanted to find out if a cordless controller uses an "USB recharge cable" for communication, if the cable is connected. Therefor I used I a high frequency detector as it will visualize any emissions. Surprise surprise surprise: You will see in the small clips, that the PS3 Controller stops emitting when the USB cable is attached, while the XBOX 360 controller continues to communicate via bluetooth. But but but, that was not the big surprise. The surprise was that both consoles, even if wireless lan was deactivated, where emitting at full strength wlan radiation. Even if the console are in standby! "Always on" - somehow in this console generation too.
Xbox 360 emission test - wireless lan always on:
XBOX 360 controller bluetooth emission test for latency optimization:
PS3 controller bluetooth emission test for latency optimization and wireless lan always on:
Back to the Rift PC. I decided to use a Blackmagic Intensity Pro PCIX card as capture card for the public project, because it is cheap and available for most users. It accepts 1280/720 at 50HZ/59,94HZ/60HZ, 1080p at 23,94hz/24hz/30hz, 1080i at 50hz/59,94hz/60hz, PAL/NTSC ans their progressive versions. Most important is that the card is supported (like newer/better Blackmagic cards) by the Steroscopic Player. Below are a few diagrams of how to playback sources in real time on an Rift PC. The HDMI splitter in the beginning is not an must, but it must work without delay. The Blackmagic produces "black screens" if a supported resolutions comes in unsupported timings. So with the HDMI splitter you can add a control monitor and always see your source. Even if Blackmagic works as intended, you will not need to wear your Rift for configuration.
The blackmagic will capture video/svideo/component at the above mentioned resolutions. Component can be converted to Euro Scart RGB via a transcoder - this conversion is lossless because it is a colour space conversion. So the Rift PC will accept all those mentioned sources. I will not go into detail with old SD and HD sources, but directly step to stereoscopic input of NVidia, Tridef and IZ3d.
The input of Tridef/IZ3D PC´s can go directy to the Rift PC and it´s Blackmagic with 720p60 SBS! 60fps per eye today via HDMI! Well, if you source PC can deliver them to the Rift PC. So you render the game including head tracking on the source PC - and then capture it and realtime rift it to the format of the Oculus Rift on your head.
Tridef/IZ3D stereoscopic 3d 720p60SBS as source - HDMI Splitter - HDMI to Blackmagic Intensity PCIx - Stereoscopic Player/Rift PC - output at 1280x800p6ohz "Side by Side" with Rift geometry.
Nvidia 3D is a little bit more tricky. Again you can directly connect the Nvidia source PC to the Rift PC and capture frame sequential at 720p60 (30hz per eye and depending on your PC also 30 frames per eye) via HDMI. Or a capture card capable of 120hz or a Lumagen Radiance....
I hesitated to link videos captured with the Rift PC because of several reasons. First, I do not have a Rift and can not adjust 3D settings and the Stereoscopic Player. I even found a way to increase the low separation in PS3 stereoscopic games, a good deal more than you can see in Uncharted. But that is of no value without a Rift for control. So when you watch the videos, I need feedback if what I did results in a correct Rift format - I can not adapt it to comfortable viewing. My second reason to hesitate with public videos is that this Rift PC is optimized for realtime playback. As you see in the camera video, playback is smooth - even at 60hz frame sequential. But when I capture with the lame but low latency Rift PC, the videos do stutter. So please keep that in mind. I even was forced to deactivate sound recording to unlock capacities.
Thanks again to Matthias for the helping camera hand, and his massive XBOX collection.
NVIDIA stereoscopic test application 720p60 framesequential realtime rifted for Oculus Rift
Tridef does Trine and then realtime rifted for Oculus Rift
Nvidia does Dreamfall and then is realtime rifted for Oculus Rift
IZ3D does Trine and then realtime rifted for Oculus Rift
PS3 and Xbox 360 3D (!) sereoscopic (!) gaming on the Oculus Rift:
Two points are left:
First - to the person giving out free Oculus Rifts to researching persons:
I could need a Rift for improving this solutions and realizing others. Also, as soon as HD-Rift is available, I could adapt the real time rifting solutions to the new panel.
Second - to the staff manager of "Team Oculus Dallas":
Consider this project the attachment of my application. My Job is to identify problems in all (hardware/physical) fields, become an expert and solve them - including the writing of unassailable patents (if patents are possible or necessary.)
Thanks for reading and best regards from Munich, Germany