Reverse Engineering 4K Networking - AVS Forum
Forum Jump: 
Closed Thread
 
Thread Tools
post #1 of 499 Old 12-30-2001, 11:57 PM - Thread Starter
Member
 
mlinehan's Avatar
 
Join Date: Dec 2001
Location: Columbus Ohio
Posts: 42
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Ha! I've done it! :D

I've succeded in convincing my 4080 that my PC is another 4080!!

I had a small lower-case / upper-case errro in my UPnP:Alive SSDP multicasts, but I've got that fixed now!

As soon as I fixed my UPnP:Alive packet, my 4080 attempted to retrieve "Device_Descr.xml" from my PC. You only get one shot at this. If the PC fails to deliver the requested xml file, the 4080 won't ask again. Had to keep rebooting the 4080 until I got it right.

So I hacked up a quick www server in my ReplayPC application and fed the 4080 a "Device_Descr.xml" that describes my PC as a 4080 called "ReplayPC".

Sure enough, the replay guide now allows me to select a remote 4080 called "ReplayPC". When I select the ReplayPC ReplayGuide, the 4080 sends an HHTP request to my PC:

http://192.168.0.4/http_replay_guide...4080K000000000

Note, the serial number in the above request is the serial number of the 4K requesting guide data. I've zeroed it out here, the 4080 does not seem to care.

Point your www browser at the above url (correct the ip address as required) and the 4K will dump the entire ReplayGuide through the HTTP connection. InternetExplorer does not understand this data, so it chokes on it. But if you watch with a packet sniffer, it's easy to see that the entire contents of the ReplayGuide is in the transmission. (10.5KB of data in my case)

Well time for some sleep, and then back to work.

Anyone else out there willing to help? ;)

mlinehan
mlinehan is offline  
Sponsored Links
Advertisement
 
post #2 of 499 Old 12-31-2001, 01:22 AM
Senior Member
 
markus98's Avatar
 
Join Date: Dec 2001
Location: Tucson, Arizona
Posts: 325
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
...192.168.0.x is all private IP address
space and not globally routable.

You're behind a network address translation
device (NAT gateway) and possibly a multiple
address to one using port address translation
(PAT) such as a D-LINK DI713P or LinkSYS BESWFR4
or whatever it is ;-)

If you truly want us to connect to your 4080,
you need to
1. Give us the globally routable address
2. If you're using NAT or PAT, make your router
map incoming connects on port 80 on the
extransl address to port 80 on the .4 address.

Cheers, and happy new year

M

markus98 is offline  
post #3 of 499 Old 12-31-2001, 01:52 AM - Thread Starter
Member
 
mlinehan's Avatar
 
Join Date: Dec 2001
Location: Columbus Ohio
Posts: 42
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Why would I wan't you to connect to my 4080???

I'm telling YOU how to get the ReplayGuide from YOUR 4xxx.

The hope being that toghther, we can reverse engineer the guide data, and then move onward to reverse enginerring how one 4K unit requests an mpeg stream from another 4K unit.

The end goal being to write a PC application that can retrieve desired mpeg files from a 4K at will.

Understand now?

mlinehan
mlinehan is offline  
post #4 of 499 Old 12-31-2001, 01:53 AM
Member
 
q3ded's Avatar
 
Join Date: Dec 2001
Location: Cupertino, CA
Posts: 70
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Dude, sweeeeet! I was parsing the data right from My ReplayTV, which of course, I was worried about being sued over. This way I could negate them completely and have more info than I need.

So much for sleep tonight. You rock!

Cheers,
Chad


____________
Share your shows, only at:
Planet Replay

Share your DVDs, now testing at:
DVD Jones
q3ded is offline  
post #5 of 499 Old 12-31-2001, 09:22 AM
Advanced Member
 
dkan24's Avatar
 
Join Date: Oct 2001
Location: New York, NY
Posts: 751
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
mlinehan,

very awesome! Even with my non-engineering background, that seems like a pretty big breakthrough!

keep up the great work- and let's make our 4000's the absolute best machine ever!
dkan24 is offline  
post #6 of 499 Old 12-31-2001, 11:01 AM - Thread Starter
Member
 
mlinehan's Avatar
 
Join Date: Dec 2001
Location: Columbus Ohio
Posts: 42
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Well I now have my ReplayPC program retrieving the ReplayGuide from my 4080 at start up and saving it to disk.

It then sends out some UPnP:Alive packets to get the attention of my 4080.

The 4080 then requests Device_Descr.xml from my PC, and I supply it one.

At this point my 4080 thinks my PC is another 4080 on the local network called ReplayPC.

Next I tell my 4080 to access the ReplayPC ReplayGuide

The 4080 then request the guide from ReplayPC, and my program then feeds the saved guide file back to my 4080, and the 4080 dutifilly allows me to select a remote program for streaming playback. (I'll work on decoding this file later)

Now things start to get interesting. The 4080 then issues a request for the following url to my PC:

/httpfs-fstat?name="/"

I've used i.e. to probe what the proper response is. But I'm a bit confused on where to go from here at the moment. I don't think my PC program is properly emulating the response just yet.

Time for a lunch break, then I'll get back on it.

Anyone else out there ready to report any sucesses?

mlinehan
mlinehan is offline  
post #7 of 499 Old 12-31-2001, 02:18 PM - Thread Starter
Member
 
mlinehan's Avatar
 
Join Date: Dec 2001
Location: Columbus Ohio
Posts: 42
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
A summary of what I know so far.

The 4K's find each other on the local network via Universal Plug and Play. This involves SSDP multicast packets to 239.255.255.250, udp port 1900. These multicast packets contian an HTTP url with wich to retrived the UPnP Device description XML file: "n.n.n.n/Device_Descr.xml"

When one 4K first becomes aware of another 4K, it request retrieves the "Device_Descr.xml" to discover the "Friendly Name" of the new 4K on the network.

Note: The "Device_Descr.xml" file that the 4K is serving up is NOT complient with the UPnP standard. UPnP requires that the decripton file enumerate the available services, and provide directions on how to use the SOAP remote procedure call mechinisim to access them. These are MISSING!!!!

At this point, the 4K ReplayGuide will allow access to the new 4K on the network.

As soon as a remote ReplayGuide is accessed by a 4K user, the 4K sends a HTTP request to the remote 4K to retrieve the Remote ReplayGuide. The url is n.n.n.n/http_replay_guide-get_snapshot?guide_file_name=0&serial_no=RTV4080K0000000000 (The serial number is that of the requesting 4K unit, I don't know what name is. 0 seems to work consistantly)

The remote 4K unit then dumps it's entire ReplayGuide back in one lump sum. I do not yet know the format for this data. There appears to be both binary and null terminated text mixed in. For now, I just write this to disk, and then echo it back when my 4K asks my PC for it's guide file.

At this point, the 4K user can browse the remote 4K's replay guide. If the user select "Play" for a remote program, the 4K then sends a request for: n.n.n.n/httpfs-fstat?name="/" to the remote 4K.

The correct response appears to be
"0\
type=d\
"

Next the 4K sends a request for: n.n.n.n/httpfs-fstat?name="/Video" to the remote 4K

The correct response appears to be:
"80820024\
"

One time, and one time only, I saw the 4K request: n.n.n.n/httpfs-fstat?name="/Video/1009609197.mpg"

I do not know the correct response, and I have not been able to repeate the feat. But I would assume that this is the actual name of the mpg file I'm attempting to stream!!! I do not see an obvious connection between the mpg file name and data in the ReplayGuide file yet.

More research will be needed.

I need to take another break.
Is anyone else out there working this?
I could use some support here. I really don't want to be a one man reverse engineering crew!

Happy hunting!!! ;)

mlinehan
mlinehan is offline  
post #8 of 499 Old 12-31-2001, 02:37 PM
AVS Special Member
 
toots's Avatar
 
Join Date: Aug 2000
Location: Bedford, NH USA
Posts: 4,084
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
/Video is where the mpeg partition is rooted in the... UPnP??? system.

Looking at the replay's config file /sys2/etc/httpfs-export, you see three different roots being exported: the program guide, the Photo partition (apparently named "storage," exported as "Photo") and the mpeg partition, exported as "Video."

The name should indeed be the filename that the program appears under in the mpeg partition.

Everyone sucks more than everyone else
host of the Curmudgeon Message Boards
toots is offline  
post #9 of 499 Old 12-31-2001, 02:44 PM
AVS Special Member
 
toots's Avatar
 
Join Date: Aug 2000
Location: Bedford, NH USA
Posts: 4,084
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
ps - I'm betting that the 100xxxxxxx filename is probably derived from some sort of timestamp, for instance when the program was created. Probably a unix date/timestamp. For instance, with the example you gave, backtranslating from unix timestamp gives December 29, 2001, 1:59:57.

Everyone sucks more than everyone else
host of the Curmudgeon Message Boards
toots is offline  
post #10 of 499 Old 12-31-2001, 02:48 PM
AVS Special Member
 
toots's Avatar
 
Join Date: Aug 2000
Location: Bedford, NH USA
Posts: 4,084
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally posted by mlinehan
Next the 4K sends a request for: n.n.n.n/httpfs-fstat?name="/Video" to the remote 4K

The correct response appears to be:
"80820024\
"
Interesting... last night, while horsing around with WS_PING (the most handy tool I have for flinging HTTP requests around without using a browser), I was doing a n.n.n.n/httpfs-ls?name="/Video", and the sole response I got back was that "80820024\
" (from memory... if not that, something that looked very much like it).

Wonder what that number means?

Everyone sucks more than everyone else
host of the Curmudgeon Message Boards
toots is offline  
post #11 of 499 Old 12-31-2001, 02:50 PM - Thread Starter
Member
 
mlinehan's Avatar
 
Join Date: Dec 2001
Location: Columbus Ohio
Posts: 42
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Thanks Toots!

The filename is indeed the unix time stamp.

Said mpg file was recorded on:
12/29/2001 at 2:00AM

The answer was right in front of my face and I didn't see it.
For shame, I have used Unix Timestamps like this myself several times in the past. I should have seen it!

Thanks! ;)

mlinehan
mlinehan is offline  
post #12 of 499 Old 12-31-2001, 02:55 PM - Thread Starter
Member
 
mlinehan's Avatar
 
Join Date: Dec 2001
Location: Columbus Ohio
Posts: 42
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
As I understand it, the RPTV's are running VXWorks, a unix like real time OS.

Could the 80820024 be a representation of the file access rights in decimal?

I've never been good unix access rights, owner, group, other, and all that.

mlinehan
mlinehan is offline  
post #13 of 499 Old 01-01-2002, 09:40 AM
 
centaur's Avatar
 
Join Date: Dec 2001
Posts: 319
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Well I'll be damned...

Nice work Mlinehan!

Stepping back, it looks like the HTTP FileService is trying to stat (access) the Mpeg partition, and speculating, in that access request should be some kind of hash for the desired .mpg file, which the RP is not sending in your example. Looks like it accesses the drive, then the part, then should come the file request.

Wow, I can't believe this is not encrypted.

They is gwine lose dey lawsuit...

Better lock em down boys, to prevent a feature-ectomy.

I'm envisioning a "TV Engine"(TM ), with IR/UHF in/out on ttyS1, firewire tapped into the sat receiver, and video card going directly to the projector!

Me, I'm still waitin' for my 4k. (snif)TM
centaur is offline  
post #14 of 499 Old 01-01-2002, 03:57 PM - Thread Starter
Member
 
mlinehan's Avatar
 
Join Date: Dec 2001
Location: Columbus Ohio
Posts: 42
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
The url to request the mpg file is:
n.n.n.n/httpfs-readfile?pos=0&name="/Video/xxxxxxxx.mpg"

But you can't do this from a normal www browser. The HTTP request must contain an "Authorization: Basic aaaaaaaaaaaaaaaaaaaaaaaa" option or the 4K won't reply.

I leave it as an exercise to the reader to discover how to coax the "aaaaa" auth code from the 4K

The file is being transfered via HTTP with chunk transfer encoding. I didn't understand chunk transfer encoding, so my first wack resulted in a 1.7GB file on the PC. But the file was not useable because the HTTP protocol headers used for chunk encoding were written into the file.

I've since updated my program to understand chunk encoding, and I have another file transfer underway!!!!

BTW: each xxxxxxx.mpg file has an associated xxxxx.ndx file. The 4K's transfer the nxd file before requesting the mpg file.

Anyone have any ideas what the ndx file may be??? For my 1.7GB mpg, the ndx is 241KB.

L8tr
L8tr

mlinehan
mlinehan is offline  
post #15 of 499 Old 01-01-2002, 06:45 PM - Thread Starter
Member
 
mlinehan's Avatar
 
Join Date: Dec 2001
Location: Columbus Ohio
Posts: 42
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
OK, I'm packing it in for a while since I have to resume my day job! ;) At least until next weekend.

Even though I won't be working on my ReplayPC program for a while, I'm willing to answer others questions. E-mail me....

I have retrieved a 1 hour medium quality file from my 4080, and I think that I've properly stripped out all the HTTP protocol stuff.

I copied the code from extract_rtv that corrects the mpg header and put it in my program, but windows media player still insists that it does not have a codec with which to play the file.

Here is a hex dump of the first 500H bytes of my resulting mpg file. Can someone who knows a bit about mpg files take a look at this and let me know if they see anything incorrect!

00 00 01 BA 44 00 04 00 04 01 00 A5 03 F8 00 00
01 BB 00 0C 80 52 81 06 E1 FF E0 E0 E0 C0 C0 20
00 00 01 BC 00 12 E1 FF 00 00 00 08 02 E0 00 00
03 C0 00 00 AA 58 7B F9 00 00 01 BA 44 00 04 00
04 01 00 A5 03 F8 00 00 01 BB 00 0C 80 52 81 06
E1 FF E0 E0 50 C0 C0 50 00 00 01 C0 02 AA 81 80
07 21 00 01 8C B3 FF FF FF FD B4 04 66 66 33 44
66 45 44 44 44 44 44 6E 46 DB 48 90 00 01 22 00
00 00 AA AA 5A A5 54 5F 5F AA AA 8F 3D 55 5F 7E
28 A4 82 48 23 8E 49 27 96 79 65 8E 58 E5 92 5A
24 9E 39 E3 92 59 25 96 39 63 9E 49 E4 92 49 A6
9A 65 96 5E DA 64 F6 D3 27 8C 9F AC 64 FD 69 AE
9A D1 94 65 CB 1B 96 35 FE 2D D6 55 96 BA AE 9B
26 C8 C1 30 4E 83 A2 25 12 66 D9 B3 51 A9 58 AC
76 ED 29 28 51 AB 17 20 58 B9 00 AC 31 85 61 8D
4E B4 EA AD AB 6E 96 DB 2D B7 22 F0 B1 AC 65 C1
70 B8 AD 09 F2 7C 94 25 01 B0 DC 1D 07 57 AB CA
35 1B E7 CD 6E 8D 0D 7C C6 AB E6 35 4B 64 70 5B
23 8A A5 AA 56 01 80 54 78 A8 F1 7E 6B 73 24 C9
4A 92 A2 2C 8B 58 96 21 A0 B9 49 A4 E0 F8 3C B4
5A 8D C6 C8 91 73 8A 14 9C 4C B9 A2 65 CC E7 09
57 38 4A DE AD EA A1 88 67 64 AE C9 4B F3 D3 3E
4D 0C 1F 07 B6 8D A2 10 84 2A 4D 1B 4D A6 2F 15
F2 29 12 69 34 E1 C4 21 A3 41 A2 8C 31 14 61 8A
74 47 52 A2 39 65 96 59 BF 6F E8 EB D1 F7 5D 5E
1D A2 6A E0 C8 31 C5 6F 21 09 8C 1D 1B AA 55 26
D9 B6 51 A8 E3 51 A4 08 1A D5 2A 68 DB 04 06 D7
E0 69 57 07 4A B8 28 DA 8D AC 03 01 77 B2 EF 62
34 CE CD DB 76 93 24 D0 64 18 BA 24 33 7C DE 5B
2D AD 4B 9C 4A 24 94 4A 56 AD 6B 53 A6 6C 38 A3
61 C5 1A E1 69 97 0B 4C 50 B5 0B 3A 0E 85 0E 4A
1C 87 07 52 45 11 42 BC AF 71 DC 75 91 01 B2 AC
A8 64 31 63 58 E5 D2 E4 62 31 E3 C8 C5 4A 9A 14
2A C0 A1 66 14 87 72 A4 3F 8A 8A A8 A8 72 1C CD
D4 93 A8 D9 B2 CA 34 8D 37 8D E3 88 E2 39 0E 9A
38 8E 17 0B 8A BA AE DA 6D 44 A2 50 9E C1 AF 9F
1A 62 B9 73 15 CB 4B 0A EA 58 57 75 DF 5D CD 33
4E 10 84 21 08 59 66 CC C3 32 C1 B0 6B 1A C4 7A
1E 96 D5 B6 4D 26 46 91 A2 89 45 4C A6 1A 35 8C
7C FA 8F E6 E3 7F 37 10 C7 F8 46 BF C3 38 13 81
0C 43 1B 6C 76 D8 AB 50 90 20 08 05 B1 6C 5A 16
8F 03 C0 6B 9A F4 3A 1B AA EA 86 43 2D C6 E4 58
A7 3A 54 A6 D0 E4 CE 87 26 86 E7 3E 37 39 F3 95
39 5A C2 B0 90 E5 21 C8 C8 47 44 85 21 36 8D 97
1D D0 9C E1 83 2C CB 92 49 03 00 C2 B5 52 95 CA
DC B9 6C 29 D3 A2 BB 90 56 5C 82 F7 2A 27 B9 51
2C B2 CB 2C 0B 03 5B 06 B6 0B 63 BB A5 A9 6A 3C
8F 2E 8B A2 20 88 B7 8D E0 44 22 DF 37 D6 BB 58
84 42 E5 CA 94 A5 49 80 00 00 01 C0 02 AA 81 80
07 21 00 01 9D 93 FF FF FF FD B4 04 33 55 33 44
45 44 55 33 33 22 33 92 49 23 92 44 89 00 00 00
00 00 AA 50 50 1A 4A 00 00 05 50 54 10 51 47 DE
7D E8 A5 7A 29 5E 89 D8 9D 96 48 25 92 09 64 76
57 61 86 58 65 A2 18 E3 9A 48 26 96 09 E6 76 79
9D 96 47 66 91 D9 A5 72 69 5C 9E 67 A8 99 F9 DD
9D D9 DC 9D CA 67 76 99 DD 99 D9 9D 30 83 08 73
01 9C C0 66 9A 69 A4 45 11 AB D5 F6 43 90 E1 56
15 5B 4D A7 43 A1 CB 96 C3 62 3F 7B 8D 22 05 F5
7D 61 F5 AA D6 B5 C1 D0 74 48 8E 77 2A F2 AE A9
43 A2 50 CB DC BD C1 50 53 7F 6F 29 42 62 04 20
43 1A 8D 36 93 56 2C 2A 95 46 71 9C 65 18 2B C2
F0 98 6A 9D F7 7D 5B 56 D2 24 6B 61 DA 1D AF 45
FB D1 7F 49 74 D6 A8 AA 1A C9 48 27 09 C5 D8 5D
88 9C 4D 3A 9D 50 A2 CB 65 9C 67 1A 9E 9E 41 D0
73 4D C2 0B 83 0B 22 27 5E C1 08 D1 8D 19 68 3D
5A 0F 58 A7 8A 6A 52 94 B5 AB 39 DA 76 B9 B3 9B
B6 9B 4E 47 23 E7 D1 38 9B 6A DA 99 A3 E0 4A 12
D4 38 E6 21 88 AE EB B9 D3 9A D8 97 89 72 E9 A4
BA 69 88 38 83 AA AA A1 E6 1C C4 11 0A 5B 65 B9
32 99 86 61 6E DD 4F 17 8D 83 62 0F 83 E9 9A 64
8E B0 AE EB BA 30 8C 3D 7A D6 AA 42 A4 29 15 0A
45 42 55 25 51 AE 6B B8 18 D2 61 18 4C D2 AB 4D
36 99 48 20 F1 E4 72 38 2D 8B 6B 42 D0 D9 36 48
9B 49 07 3F 96 15 86 5C B6 31 CD 1C D0 8E 47 23
91 D6 F1 6F 0C 53 11 C7 99 85 B5 6D 86 78 57 6D
B6 E2 74 B7 CF 8B 65 B7 11 C4 4B 52 D6 A9 AA 7A
98 AE 7B 9E B0 AC 2F 9F 10 92 8B 28 B6 C2 C5 B0
B0 C9 0C 90 DA 36 B0 23 2D 2D 6B 53 C4 C4 53 6D
B6 A3 56 24 48 DB 4E 4B 12 C2 48 92 36 6D 98 E0
D1 68 9A 24 21 08 7A F6 D6 9D C9 DC 92 15 64 85

mlinehan
mlinehan is offline  
post #16 of 499 Old 01-01-2002, 08:46 PM
Advanced Member
 
dkan24's Avatar
 
Join Date: Oct 2001
Location: New York, NY
Posts: 751
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
could the ndx file be the commercial skip information?

At some point, someone has to write a program to interpret the ndx into Womble and automatically cut out the commercials from the mpeg! :) :)

Have you tried playing the file in a software DVD player such as PowerDVD or WinDVD? I know that WinDVD will play straight mpeg2 files. If you want to send me the file, I could try opening it in any of those or Womble.
dkan24 is offline  
post #17 of 499 Old 01-01-2002, 09:22 PM
AVS Special Member
 
toots's Avatar
 
Join Date: Aug 2000
Location: Bedford, NH USA
Posts: 4,084
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I doubt that it has commercial skip information. I distinctly recall them being present on the 2000/3000 series MPEG partitions as well.

I always assumed they contained specific show content information as well as some kind of back-pointer to the replay channel database on the system partition.

Everyone sucks more than everyone else
host of the Curmudgeon Message Boards
toots is offline  
post #18 of 499 Old 01-01-2002, 09:47 PM
Advanced Member
 
FreezeFrame's Avatar
 
Join Date: Sep 2000
Location: Livermore, CA, USA
Posts: 969
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I know nothing, nothing, nothing!!!
FreezeFrame is offline  
post #19 of 499 Old 01-01-2002, 10:35 PM - Thread Starter
Member
 
mlinehan's Avatar
 
Join Date: Dec 2001
Location: Columbus Ohio
Posts: 42
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
My new years gift to the comunity, the source code for my ReplayPC application. This compiles under VC6.0, and runs on my Win2K system. You mileage may varry.

Note: The output mpg file is not correct yet, but I am getting close.

Code:
//**********************************************************************************************
//
// ReplayPC.c  Retrieve the guide file and .mpg files from a 4000 series ReplayTV via network
// WARNING - This is ALPHA software. Don't try to use this unless you know what your doing!
//
//
//  Release history:
//  v0.1    01/01/2002  First Souce Code Release, Just to let others help in this effort
//                      This code comiles with VC++ 6.0, but I've attempted to keep it as
//                      portable as possible to other operating systems. Thats why I chose
//                      to write this as a Win32 Console Application. Microsoft makes this hard!
//                      Development and testing were done on a PII-400 Win2K system     
//
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
//
// If you distribute a modified version of this program, the program MUST tell the user
// in the usage summary how the program has been modified, and who done it!
// Modified source code MUST be distributed with the modified program.
//
// Please submit patches and sugestions to mlinehan@columbus.rr.com
//
//**********************************************************************************************
#include <stdio.h>
#include <conio.h>
#include <stdlib.h>
#include <winsock2.h>

//Tell the linker to link with the winsock library
#pragma comment(lib, "ws2_32.lib")


//**********************************************************************************************
// Global Variables and Defines
//**********************************************************************************************



//**********************************************************************************************
// Function Prototypes
//**********************************************************************************************
void GetReplayGuide(char *IPAddress);
void GetMpgFile(char *IPAddress, char *FileName);
int  RecieveHTTPData(SOCKET s, char* Buffer, int Len, char Terminator);


//**********************************************************************************************
// Main Loop
//**********************************************************************************************
int main(int argc, char* argv[])
{
    WSADATA wsaData;            //Winsock Garbage
    LPVOID  Dummy       = NULL; //Winsock Garbage
    int     fGetGuide   = 0;    //Flag, true if were getting the guide file
    int     fGetFile    = 0;    //Flag, true if were getting a file
    char    *FileName   = NULL; //pointer to filename
    char    *IPAddress  = NULL; //pointer to ip address as a string
    int     i           = 0;    //temp counter

    //fire up windows sockets version 2.2
    if(WSAStartup(MAKEWORD( 2, 2 ), &wsaData ) == -1)
    {
        perror("WSAStartup() Failed\
");
        exit(1);
    }

    //Print usage summary if options are not supplied
    if(argc < 3) 
    {
        printf("ReplayPC V0.1, built on " __DATE__ " " __TIME__ "\
");
        printf("USAGE: %s <4k-ipaddress> <options>\
", argv[0]);
        printf("  -g            Retrieve Replay Guide File, Write to Disk\
");
        printf("  -e filename   Retrieve MPG File Named filename, Write to Disk\
");
        printf("EXAMPLE: %s 192.168.0.4 -g\
", argv[0]);
        printf("EXAMPLE: %s 192.168.0.4 -e 1009609197.mpg\
", argv[0]);
        return -1;
    }
        
    //scan command line arguments
    IPAddress = argv[1];
    for(i=2;i<argc;i++) 
    {
        if(argv[i][0] == '-') 
                {
            if(argv[i][1] == 'g') 
            {
                fGetGuide = 1;          
            } 
            else if(argv[i][1] == 'e') 
            {
                fGetFile = 1;
                if(((i+1)<argc) && (argv[i+1][0] != '-')) 
                {
                    i++;
                    FileName=argv[i];
                } 
            } 
            else 
            {
                fprintf(stderr,"Ignored command line option: %s\
",argv[i]);
            }
        }
    }
        
    //Do as we are instructed
    if (fGetGuide) GetReplayGuide(IPAddress);
    if (fGetFile)  GetMpgFile(IPAddress, FileName);

    //shutdown windows sockets
    WSACleanup( );

    //cleanup and exit
    return 0;
}


//**********************************************************************************************
// Retrieve the 4K Replay Guide and write to disk
//**********************************************************************************************
void GetReplayGuide(char *IPAddress)
{
    SOCKET  s;                      //A socket request the guide data with
    struct  sockaddr_in adr_dst;    //Destination IPAddress structure
    char    Buffer[1500];           //A Buffer for our work
    int     len, chunklen;          //a temp for dealing with the buffer
    FILE    *fileptr;               //a normal file pointer
    int     Done;                   //temps

    //A HHTP request for the ReplayGuide File
    char ReqGuide[]="GET /http_replay_guide-get_snapshot?guide_file_name=0&serial_no=RTV4080K0000000000 HTTP/1.1\
\
"
                    "Host: 192.168.0.4:80\
\
"
                    "Accept-Encoding: gzip\
\
"
                    "\
\
";

    //fill the destination address/port structure
    memset(&adr_dst, 0, sizeof(adr_dst));
    adr_dst.sin_family = AF_INET;
    adr_dst.sin_port = htons(80);
    adr_dst.sin_addr.s_addr = inet_addr(IPAddress);

    //get a socket to work with
    s = socket(AF_INET, SOCK_STREAM, 0);
    if (s == INVALID_SOCKET)
    {
        perror("GetGuide:socket() Failed\
");
        exit(1);
    }

    //Open the file we will use to store the guide data
    if ((fileptr = fopen("Guide.dat", "wb")) == NULL)
    {
        perror("GetGuide:fopen() Failed\
");
        exit(1);
    }

    //Initiate the connection to the 4K
    if (connect(s, (struct sockaddr*)&adr_dst, sizeof(adr_dst)) == -1)
    {
        perror("GetGuide:connect() Failed\
");
        exit(1);
    }
        
    //Tell the user were up to something
    printf("Retrieving ReplayGuide file...\
");

    //Send the request for the guide
    send(s, ReqGuide, strlen(ReqGuide), 0);

    //retrieve all the data
    Done=0;
    do
    {
        //Skip past the HTTP header junk by searching for the blank line
        do
        {
            len = RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x0A);
        }while(len != 2);

        //The next line should be a chunk lengh specifier
        len = RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x0A);
        sscanf(Buffer, "%x", &chunklen);
        printf(".", chunklen); 

        //If this chunk is non zero in length
        if (chunklen)
        {
            //Loop for chunklen bytes of data
            len=0;
            do
            {
                if (chunklen > sizeof(Buffer))
                {
                    //Recieve sizeof(Buffer) bytes of binary data
                    len = RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x00);
                    chunklen -= len;
                }
                else
                {
                    //Recieve chunklen bytes of binary data
                    len = RecieveHTTPData(s, Buffer, chunklen, 0x00);
                    chunklen = 0;
                }

                //write the data to disk
                fwrite(Buffer, len, 1, fileptr);

            }while(chunklen);

        }
        else
        {
            //chunklen is zero, signifying end of transmission
            Done=1;
        }
    }while(!Done);
        
    //all done, cleanup as we leave
    printf("\
Done, ReplayGuide written to [guide.dat].\
");
    fclose(fileptr);
    closesocket(s);
}


//**********************************************************************************************
// Retrieve a MPG file and write it to disk
//**********************************************************************************************
void GetMpgFile(char *IPAddress, char *FileName)
{
    SOCKET  s;                      //A socket request the guide data with
    struct  sockaddr_in adr_dst;    //Destination Address
    char    Buffer[1500];           //Buffer to recieve the guide data into
    int     len, chunklen;          //a temp for dealing with buffers
    FILE    *fileptr;               //a normal file pointer
    int     Done=0;                 //temp flag
    int     FirstTime=0;            //another flag
    char    ReqMpg[500];            //build the HTTP request here

    //build the HTTP request
    //NOTE: The Authorization is very important. This code works with my 4K
    //      I don't know if it will work for others....
    sprintf(ReqMpg, "GET /httpfs-readfile?pos=0&name=\\"/Video/%s\\" HTTP/1.1\
\
", FileName);
    strcat(ReqMpg,      "Authorization: Basic Uk5TQmFzaWM6QTd4KjgtUXQ=\
\
");
    strcat(ReqMpg,      "User-Agent: Replay-HTTPFS/1\
\
");
    strcat(ReqMpg,      "Host: 192.168.0.2\
\
");
    strcat(ReqMpg,      "Accept-Encoding: gzip\
\
");
    strcat(ReqMpg,      "\
\
");

    //fill the destination address/port structure
    memset(&adr_dst, 0, sizeof(adr_dst));
    adr_dst.sin_family = AF_INET;
    adr_dst.sin_port = htons(80);
    adr_dst.sin_addr.s_addr = inet_addr(IPAddress);

    //get a socket
    s = socket(AF_INET, SOCK_STREAM, 0);
    if (s == INVALID_SOCKET)
    {
        perror("GetMpgFile:socket() Failed\
");
        exit(1);
    }

    //Open the file we will use to store the response data
    if ((fileptr = fopen(FileName, "wb")) == NULL)
    {
        perror("GetMpgFile:fopen() Failed\
");
        exit(1);
    }

    //Connect to the 4K
    if (connect(s, (struct sockaddr*)&adr_dst, sizeof(adr_dst)) == -1)
    {
        perror("GetMpgFile:connect() Failed\
");
        exit(1);
    }
        
    //Tell the user were up to something
    printf("Retrieving /Video/%s...\
", FileName);

    //Send the request for the file
    send(s, ReqMpg, strlen(ReqMpg), 0);
        

    //Initialize flags
    Done=0;
    chunklen=0;
    FirstTime=1;
    do
    {
        //Skip the HTTP header stuff by searching for the blank line
        do
        {
            len = RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x0A);
        }while(len != 2);

        //This line should be a chunk lengh specifier
        len = RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x0A);
        sscanf(Buffer, "%x", &chunklen);
        printf(".", chunklen); 

        //For some bizzar reason, the fist chunk has a line feed terminated
        //number before the mpg data, skip over it
        if (FirstTime)
        {
            chunklen -= RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x0A);
        }

        //If the chunk lenght is nonzero
        if (chunklen)
        {
            //Loop for chunklen bytes of data
            len=0;
            do
            {
                if (chunklen > sizeof(Buffer))
                {
                    //Recieve sizeof(Buffer) bytes of binary data
                    len = RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x00);
                    chunklen -= len;
                }
                else
                {
                    //Recieve chunklen bytes of binary data
                    len = RecieveHTTPData(s, Buffer, chunklen, 0x00);
                    chunklen = 0;
                }

                //mpg file fixup from extract_rtv!
                if (FirstTime)
                {
                    // Write a small pack that includes a system header and a program stream map
                    // This fixes incompatibility issues with Womble MPEG-2 VCR and MProbe and
                    // makes the MPEG-2 files much more MPEG-2 compliant which apparently they
                    // don't need to be for the ReplayTV box to play them back.
                    FirstTime=0;
                    unsigned long rate_bound;
                    unsigned char pack_hdr[] = {0x00, 0x00, 0x01, 0xba,             // pack_start_code
                                                0x44, 0x00, 0x04, 0x00, 0x04, 0x01,     // system_clock_reference_base
                                                0x00, 0x6f, 0xe7,                   // program_mux_rate
                                                0xf8};                              // pack_stuffing_length
                    unsigned char sys_hdr[] =  {0x00, 0x00, 0x01, 0xbb,             // system_header_start_code
                                                0x00, 0x0c,                         // header_length
                                                0x80, 0x37, 0xf3,                   // rate_bound
                                                0x06,                               // audio_bound/CSPS_flag
                                                0xe1,                               // system_audio_lock_flag/system_video_lock_flag/video_bound
                                                0xff,                               // packet_rate_restriction_flag/reserved
                                                0xe0,                               // stream_id
                                                0xe0, 0xe0,                         // P-STD_buffer_bound_scale/P-STD_buffer_size_bound
                                                0xc0,                               // stream_id
                                                0xc0, 0x20};                        // P-STD_buffer_bound_scale/P-STD_buffer_size_bound
                    unsigned char psmap[] =        {0x00, 0x00, 0x01,                   // packet_start_code_prefix
                                                0xbc,                               // map_stream_id
                                                0x00, 0x12,                         // program_stream_map_length
                                                0xe1,                               // current_next_indicator/program_stream_map_version
                                                0xff,                               // reserved
                                                0x00, 0x00,                         // program_stream_info_length
                                                0x00, 0x08,                         // elementary_stream_map_length
                                                0x02,                               // stream_type
                                                0xe0,                               // elementary_stream_id
                                                0x00, 0x00,                         // elementary_stream_info_length
                                                0x03,                               // stream_type
                                                0xc0,                               // elementary_stream_id
                                                0x00, 0x00,                         // elementary_stream_info_length
                                                0xaa, 0x58, 0x7b, 0xf9};            // CRC_32

                    // Make sure this is really an MPEG-2 file before we mess around with it
                    if ((Buffer[0]!=0x00) || (Buffer[1]!=0x00) || (Buffer[2]!=0x01) || (Buffer[3]!=(char)0xba))
                    {
                        printf("Not an MPG file! %X %X %X %X\
", (int) Buffer[0], (int) Buffer[1], (int) Buffer[2], (int) Buffer[3]);
                        exit(1);
                    }
                                                
                    // Copy the first pack header from the Replay's MPEG file
                    // but don't copy the pack_stuffing_length since we don't
                    // want any stuffing_bytes
                    memcpy(pack_hdr, Buffer, sizeof(pack_hdr) - 1);
                    fwrite(pack_hdr,1,sizeof(pack_hdr),fileptr);

                    // Set the system header's rate_bound to the same as the
                    // pack header's program_mux_rate. Since this is a CBR
                    // MPEG-2 file we're dealing with, these values can be
                    // the same
                    rate_bound = (pack_hdr[10] << 16 | pack_hdr[11] << 8 | pack_hdr[12]) >> 2;
                    rate_bound = (rate_bound << 1) | 0x800001;
                    sys_hdr[6] = (unsigned char)(rate_bound >> 16);
                    sys_hdr[7] = (unsigned char)(rate_bound >> 8);
                    sys_hdr[8] = (unsigned char)(rate_bound & 0xff);
                    fwrite(sys_hdr,1,sizeof(sys_hdr),fileptr);

                    //Finally, write the program stream map
                    fwrite(psmap,1,sizeof(psmap),fileptr);
                }

                //write the data to disk
                fwrite(Buffer, len, 1, fileptr);

            }while(chunklen);

        }
        else
        {
            //zero lengh chunk marks end of transmission
            Done=1;
        }
    }while(!Done);
        
    //all done, cleanup as we leave
    printf("\
Done, output written to [%s]\
", FileName);
    fclose(fileptr);
    closesocket(s);
}


//**********************************************************************************************
// A helper function for dealing with HTTP chuncked data
// We have to do this when using WinSock, because WinSock sockets are NOT streams
// in a Unix system a socket is also a file descriptor allowing use of stream I/O
//**********************************************************************************************
int RecieveHTTPData(SOCKET s, char *Buffer, int Len, char Terminator)
{
    static int PacketLen=0;
    static int SrcIndex=0;
    static int DstIndex=0;
    static char PacketBuffer[1500];
    char c;

    //Destination is zero lenght each call
    DstIndex=0;
        
    if (Terminator)
    {
        //Were retrieving terminated data
        do
        {
            //Retrieve HTTP data as required into PacketBuffer[]
            if (SrcIndex == PacketLen)
            {
                PacketLen = recv(s, PacketBuffer, sizeof(PacketBuffer), 0);
                SrcIndex = 0;
            }

            //Copy a byte from PacketBuffer[SrcIndex] to Buffer[DstIndex]
            c = Buffer[DstIndex++] = PacketBuffer[SrcIndex++];
            Buffer[DstIndex]=0x00;

        }while(c != Terminator);
    }
    else
    {
        //Were retrieving binary data
        do
        {
            //Retrieve HTTP data as required into PacketBuffer[]
            if (SrcIndex == PacketLen)
            {
                PacketLen = recv(s, PacketBuffer, sizeof(PacketBuffer), 0);
                SrcIndex = 0;
            }

            //Copy a byte from PacketBuffer[SrcIndex] to Buffer[DstIndex]
            c = Buffer[DstIndex++] = PacketBuffer[SrcIndex++];

        }while(DstIndex < Len);
    }

    //Return number of bytes coppied
    return(DstIndex);
}

//Thats all she wrote!!!
//Happy hunting my friends!

mlinehan
mlinehan is offline  
post #20 of 499 Old 01-02-2002, 09:18 AM
 
centaur's Avatar
 
Join Date: Dec 2001
Posts: 319
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Very nice work, Mlinehan. Thank you.
I'll see if this'll compile on Unix libraries.

And FreezeFrame, any dorkus can say he 'knows the secret but can't reveal it', and he'll still look like a dorkus for saying it. Happy farging new year.
centaur is offline  
post #21 of 499 Old 01-02-2002, 12:06 PM
Member
 
King Of Bayonne's Avatar
 
Join Date: Dec 2001
Location: Beautiful crime-free Albany, NY
Posts: 28
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Excellent, excellent work! :)

I too have been fiddling with the UPnP angle, and like mlinehan found that the RTV4K is *not* in compliance with the UPnP spec. Don't know if that's intentional or if they just haven't finished with the code yet... maybe in a future build...?

Anyway, I was using the universal control point app from the UPnP SDK and found the non-compliance. Too bad; I was hoping that the 4K advertised a control page so that you could actually control it from a browser...

Anyway, I was diddling with mlinehan's app, and realized that I wanted a way to simply send a given string to the box using HTTP GET to, um, discover more features. :) I hacked the code to do this, and also fixed a minor bug. Updated source code will follow in my next post.

It seems that some commands require a secure connection (e.g. file get), and others don't. I found that ALL commands will
respond over secure connections, so my mods to the test program just send a command over a secure connection.

mlinehan, your Auth data works fine with my box. :) :) :)

With new tool in hand, I started issuing commands like

replaypc ww.xx.yy.zz -c /httpfs-ls?name="/"

This listed the contents of the root directory:

Photo
SnapshotDir
Video

SnapshotDir contains:

RemoteSnapshots
Snapshot-Local-1009926000

I have no idea what these are...

Photo contains:

Test Folder

which is the name of my test folder (surprise!) for photo viewing.

Retrieving a folder list for Test Folder (had to escape the space by using %20 in the request string) listed the JPEGs I had stored on the machine.

Listing the Video directory yielded:

tmp
1009753197.mpg
1009753197.ndx
1009918797.mpg
1009918797.ndx

These correspond to the two shows I have recorded. Not sure what "tmp" is.

More coming later...

KOB
King Of Bayonne is offline  
post #22 of 499 Old 01-02-2002, 12:10 PM
Member
 
King Of Bayonne's Avatar
 
Join Date: Dec 2001
Location: Beautiful crime-free Albany, NY
Posts: 28
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
As promised, the updated souce code for mlinehan's nifty little utility follows...

Built correctly for me using VC6 SP4 on WinXP Pro.


KOB




// ************************************************************ **********************************
//
// ReplayPC.c Retrieve the guide file and .mpg files from a 4000 series ReplayTV via network
// WARNING - This is ALPHA software. Don't try to use this unless you know what your doing!
//
//
// Release history:
// v0.1 01/01/2002 First Souce Code Release, Just to let others help in this effort
// This code comiles with VC++ 6.0, but I've attempted to keep it as
// portable as possible to other operating systems. Thats why I chose
// to write this as a Win32 Console Application. Microsoft makes this hard!
// Development and testing were done on a PII-400 Win2K system
//
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
//
// If you distribute a modified version of this program, the program MUST tell the user
// in the usage summary how the program has been modified, and who done it!
// Modified source code MUST be distributed with the modified program.
//
// Please submit patches and sugestions to mlinehan@columbus.rr.com
//
// ************************************************************ **********************************
#include <stdio.h>
#include <conio.h>
#include <stdlib.h>
#include <winsock2.h>

//Tell the linker to link with the winsock library
#pragma comment(lib, "ws2_32.lib")


// ************************************************************ **********************************
// Global Variables and Defines
// ************************************************************ **********************************



// ************************************************************ **********************************
// Function Prototypes
// ************************************************************ **********************************
void GetReplayGuide(char *IPAddress);
void GetMpgFile(char *IPAddress, char *FileName);
void DoCommand(char *IPAddress, char *Command);
int RecieveHTTPData(SOCKET s, char* Buffer, int Len, char Terminator);


// ************************************************************ **********************************
// Main Loop
// ************************************************************ **********************************
int main(int argc, char* argv[])
{
WSADATA wsaData; //Winsock Garbage
LPVOID Dummy = NULL; //Winsock Garbage
int fGetGuide = 0; //Flag, true if were getting the guide file
int fGetFile = 0; //Flag, true if were getting a file
int fDoCommand = 0; //Flag, true if were sending a command
char *FileName = NULL; //pointer to filename
char *IPAddress = NULL; //pointer to ip address as a string
char *Command = NULL; //pointer to command string
int i = 0; //temp counter

//fire up windows sockets version 2.2
if(WSAStartup(MAKEWORD( 2, 2 ), &wsaData ) == -1)
{
perror("WSAStartup() Failed\
");
exit(1);
}

//Print usage summary if options are not supplied
if(argc < 3)
{
printf("ReplayPC V0.1, built on " __DATE__ " " __TIME__ "\
");
printf("USAGE: %s <4k-ipaddress> <options>\
", argv[0]);
printf(" -g Retrieve Replay Guide File, Write to Disk\
");
printf(" -e filename Retrieve MPG File Named filename, Write to Disk\
");
printf(" -c \\"command\\"' Send a command string using HTTP GET\
");
printf("EXAMPLE: %s 192.168.0.4 -g\
", argv[0]);
printf("EXAMPLE: %s 192.168.0.4 -e 1009609197.mpg\
", argv[0]);
printf("EXAMPLE: %s 192.168.0.4 -c \\"/httpfs-readfile?pos=0&name=\\"/Video/1009609343.mpg\\"\\"\
", argv[0]);
return -1;
}

//scan command line arguments
IPAddress = argv[1];
for(i=2;i<argc;i++)
{
if(argv[i][0] == '-')
{
if(argv[i][1] == 'g')
{
fGetGuide = 1;
}
else if(argv[i][1] == 'e')
{
fGetFile = 1;
if(((i+1)<argc) && (argv[i+1][0] != '-'))
{
i++;
FileName = argv[i];
}
}
else if(argv[i][1] == 'c')
{
fDoCommand = 1;
if(((i+1)<argc) && (argv[i+1][0] != '-'))
{
i++;
Command = argv[i];
}
} else
{
fprintf(stderr,"Ignored command line option: %s\
",argv[i]);
}
}
}

//Do as we are instructed
if (fGetGuide) GetReplayGuide(IPAddress);
if (fGetFile) GetMpgFile(IPAddress, FileName);
if (fDoCommand) DoCommand(IPAddress, Command);

//shutdown windows sockets
WSACleanup( );

//cleanup and exit
return 0;
}


// ************************************************************ **********************************
// Retrieve the 4K Replay Guide and write to disk
// ************************************************************ **********************************
void GetReplayGuide(char *IPAddress)
{
SOCKET s; //A socket request the guide data with
struct sockaddr_in adr_dst; //Destination IPAddress structure
char Buffer[1500]; //A Buffer for our work
int len, chunklen; //a temp for dealing with the buffer
FILE *fileptr; //a normal file pointer
int Done; //temps

//A HHTP request for the ReplayGuide File
char ReqGuide[]="GET /http_replay_guide-get_snapshot?guide_file_name=0&serial_no=RTV4080K0000000000 HTTP/1.1\
\
"
"Host: 192.168.0.4:80\
\
"
"Accept-Encoding: gzip\
\
"
"\
\
";

//fill the destination address/port structure
memset(&adr_dst, 0, sizeof(adr_dst));
adr_dst.sin_family = AF_INET;
adr_dst.sin_port = htons(80);
adr_dst.sin_addr.s_addr = inet_addr(IPAddress);

//get a socket to work with
s = socket(AF_INET, SOCK_STREAM, 0);
if (s == INVALID_SOCKET)
{
perror("GetGuide:socket() Failed\
");
exit(1);
}

//Open the file we will use to store the guide data
if ((fileptr = fopen("Guide.dat", "wb")) == NULL)
{
perror("GetGuide:fopen() Failed\
");
exit(1);
}

//Initiate the connection to the 4K
if (connect(s, (struct sockaddr*)&adr_dst, sizeof(adr_dst)) == -1)
{
perror("GetGuide:connect() Failed\
");
exit(1);
}

//Tell the user were up to something
printf("Retrieving ReplayGuide file...\
");

//Send the request for the guide
send(s, ReqGuide, strlen(ReqGuide), 0);

//retrieve all the data
Done=0;
do
{
//Skip past the HTTP header junk by searching for the blank line
do
{
len = RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x0A);
}while(len != 2);

//The next line should be a chunk lengh specifier
len = RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x0A);
sscanf(Buffer, "%x", &chunklen);
printf(".", chunklen);

//If this chunk is non zero in length
if (chunklen)
{
//Loop for chunklen bytes of data
len=0;
do
{
if (chunklen > sizeof(Buffer))
{
//Recieve sizeof(Buffer) bytes of binary data
len = RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x00);
chunklen -= len;
}
else
{
//Recieve chunklen bytes of binary data
len = RecieveHTTPData(s, Buffer, chunklen, 0x00);
chunklen = 0;
}

//write the data to disk
fwrite(Buffer, len, 1, fileptr);

}while(chunklen);

}
else
{
//chunklen is zero, signifying end of transmission
Done=1;
}
}while(!Done);

//all done, cleanup as we leave
printf("\
Done, ReplayGuide written to [guide.dat].\
");
fclose(fileptr);
closesocket(s);
}


// ************************************************************ **********************************
// Retrieve a MPG file and write it to disk
// ************************************************************ **********************************
void GetMpgFile(char *IPAddress, char *FileName)
{
SOCKET s; //A socket request the guide data with
struct sockaddr_in adr_dst; //Destination Address
char Buffer[1500]; //Buffer to recieve the guide data into
int len, chunklen; //a temp for dealing with buffers
FILE *fileptr; //a normal file pointer
int Done=0; //temp flag
int FirstTime=0; //another flag
char ReqMpg[500]; //build the HTTP request here

//build the HTTP request
//NOTE: The Authorization is very important. This code works with my 4K
// I don't know if it will work for others....
sprintf(ReqMpg, "GET /httpfs-readfile?pos=0&name=\\"/Video/%s\\" HTTP/1.1\
\
", FileName);
strcat(ReqMpg, "Authorization: Basic Uk5TQmFzaWM6QTd4KjgtUXQ=\
\
");
strcat(ReqMpg, "User-Agent: Replay-HTTPFS/1\
\
");
strcat(ReqMpg, "Host: 192.168.0.2\
\
");
strcat(ReqMpg, "Accept-Encoding: gzip\
\
");
strcat(ReqMpg, "\
\
");

//fill the destination address/port structure
memset(&adr_dst, 0, sizeof(adr_dst));
adr_dst.sin_family = AF_INET;
adr_dst.sin_port = htons(80);
adr_dst.sin_addr.s_addr = inet_addr(IPAddress);

//get a socket
s = socket(AF_INET, SOCK_STREAM, 0);
if (s == INVALID_SOCKET)
{
perror("GetMpgFile:socket() Failed\
");
exit(1);
}

//Open the file we will use to store the response data
if ((fileptr = fopen(FileName, "wb")) == NULL)
{
perror("GetMpgFile:fopen() Failed\
");
exit(1);
}

//Connect to the 4K
if (connect(s, (struct sockaddr*)&adr_dst, sizeof(adr_dst)) == -1)
{
perror("GetMpgFile:connect() Failed\
");
exit(1);
}

//Tell the user were up to something
printf("Retrieving /Video/%s...\
", FileName);

//Send the request for the file
send(s, ReqMpg, strlen(ReqMpg), 0);


//Initialize flags
Done=0;
chunklen=0;
FirstTime=1;
do
{
//Skip the HTTP header stuff by searching for the blank line
do
{
len = RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x0A);
}while(len != 2);

//This line should be a chunk lengh specifier
len = RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x0A);
sscanf(Buffer, "%x", &chunklen);
printf(".", chunklen);

//For some bizzar reason, the fist chunk has a line feed terminated
//number before the mpg data, skip over it
if (FirstTime)
{
chunklen -= RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x0A);
}

//If the chunk lenght is nonzero
if (chunklen)
{
//Loop for chunklen bytes of data
len=0;
do
{
if (chunklen > sizeof(Buffer))
{
//Recieve sizeof(Buffer) bytes of binary data
len = RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x00);
chunklen -= len;
}
else
{
//Recieve chunklen bytes of binary data
len = RecieveHTTPData(s, Buffer, chunklen, 0x00);
chunklen = 0;
}

//mpg file fixup from extract_rtv!
if (FirstTime)
{
// Write a small pack that includes a system header and a program stream map
// This fixes incompatibility issues with Womble MPEG-2 VCR and MProbe and
// makes the MPEG-2 files much more MPEG-2 compliant which apparently they
// don't need to be for the ReplayTV box to play them back.
unsigned long rate_bound;
unsigned char pack_hdr[] = {0x00, 0x00, 0x01, 0xba, // pack_start_code
0x44, 0x00, 0x04, 0x00, 0x04, 0x01, // system_clock_reference_base
0x00, 0x6f, 0xe7, // program_mux_rate
0xf8}; // pack_stuffing_length
unsigned char sys_hdr[] = {0x00, 0x00, 0x01, 0xbb, // system_header_start_code
0x00, 0x0c, // header_length
0x80, 0x37, 0xf3, // rate_bound
0x06, // audio_bound/CSPS_flag
0xe1, // system_audio_lock_flag/system_video_lock_flag/video_bound
0xff, // packet_rate_restriction_flag/reserved
0xe0, // stream_id
0xe0, 0xe0, // P-STD_buffer_bound_scale/P-STD_buffer_size_bound
0xc0, // stream_id
0xc0, 0x20}; // P-STD_buffer_bound_scale/P-STD_buffer_size_bound
unsigned char psmap[] = {0x00, 0x00, 0x01, // packet_start_code_prefix
0xbc, // map_stream_id
0x00, 0x12, // program_stream_map_length
0xe1, // current_next_indicator/program_stream_map_version
0xff, // reserved
0x00, 0x00, // program_stream_info_length
0x00, 0x08, // elementary_stream_map_length
0x02, // stream_type
0xe0, // elementary_stream_id
0x00, 0x00, // elementary_stream_info_length
0x03, // stream_type
0xc0, // elementary_stream_id
0x00, 0x00, // elementary_stream_info_length
0xaa, 0x58, 0x7b, 0xf9}; // CRC_32

FirstTime=0;

// Make sure this is really an MPEG-2 file before we mess around with it
if ((Buffer[0]!=0x00) || (Buffer[1]!=0x00) || (Buffer[2]!=0x01) || (Buffer[3]!=(char)0xba))
{
printf("Not an MPG file! %X %X %X %X\
", (int) Buffer[0], (int) Buffer[1], (int) Buffer[2], (int) Buffer[3]);
exit(1);
}

// Copy the first pack header from the Replay's MPEG file
// but don't copy the pack_stuffing_length since we don't
// want any stuffing_bytes
memcpy(pack_hdr, Buffer, sizeof(pack_hdr) - 1);
fwrite(pack_hdr,1,sizeof(pack_hdr),fileptr);

// Set the system header's rate_bound to the same as the
// pack header's program_mux_rate. Since this is a CBR
// MPEG-2 file we're dealing with, these values can be
// the same
rate_bound = (pack_hdr[10] << 16 | pack_hdr[11] << 8 | pack_hdr[12]) >> 2;
rate_bound = (rate_bound << 1) | 0x800001;
sys_hdr[6] = (unsigned char)(rate_bound >> 16);
sys_hdr[7] = (unsigned char)(rate_bound >> 8);
sys_hdr[8] = (unsigned char)(rate_bound & 0xff);
fwrite(sys_hdr,1,sizeof(sys_hdr),fileptr);

//Finally, write the program stream map
fwrite(psmap,1,sizeof(psmap),fileptr);
}

//write the data to disk
fwrite(Buffer, len, 1, fileptr);

}while(chunklen);

}
else
{
//zero lengh chunk marks end of transmission
Done=1;
}
}while(!Done);

//all done, cleanup as we leave
printf("\
Done, output written to [%s]\
", FileName);
fclose(fileptr);
closesocket(s);
}


// ************************************************************ **********************************
// Issue a command to the 4K using HTTP GET
// ************************************************************ **********************************
void DoCommand(char *IPAddress, char *Command)
{
SOCKET s; //A socket request the guide data with
struct sockaddr_in adr_dst; //Destination IPAddress structure
char Buffer[1500]; //A Buffer for our work
int len, chunklen; //a temp for dealing with the buffer
FILE *fileptr; //a normal file pointer
int Done; //temps

//A HHTP request for the command, in two parts
char Req1[]="GET ";

char Req2[]=" HTTP/1.1\
\
"
"Authorization: Basic Uk5TQmFzaWM6QTd4KjgtUXQ=\
\
"
"User-Agent: Replay-HTTPFS/1\
\
"
"Host: 192.168.0.4:80\
\
"
"Accept-Encoding: gzip\
\
"
"\
\
";

//fill the destination address/port structure
memset(&adr_dst, 0, sizeof(adr_dst));
adr_dst.sin_family = AF_INET;
adr_dst.sin_port = htons(80);
adr_dst.sin_addr.s_addr = inet_addr(IPAddress);

//get a socket to work with
s = socket(AF_INET, SOCK_STREAM, 0);
if (s == INVALID_SOCKET)
{
perror("DoCommand:socket() Failed\
");
exit(1);
}

//Open the file we will use to store the result data
if ((fileptr = fopen("result.dat", "wb")) == NULL)
{
perror("DoCommand:fopen() Failed\
");
exit(1);
}

//Initiate the connection to the 4K
if (connect(s, (struct sockaddr*)&adr_dst, sizeof(adr_dst)) == -1)
{
perror("DoCommand:connect() Failed\
");
exit(1);
}

//Tell the user were up to something
printf("Sending command \\"%s\\"...\
", Command);

//Send the request for the guide
send(s, Req1, strlen(Req1), 0);
send(s, Command, strlen(Command), 0);
send(s, Req2, strlen(Req2), 0);

//retrieve all the data
Done=0;
do
{
//Skip past the HTTP header junk by searching for the blank line
do
{
len = RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x0A);
}while(len != 2);

//The next line should be a chunk lengh specifier
len = RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x0A);
sscanf(Buffer, "%x", &chunklen);
printf(".", chunklen);

//If this chunk is non zero in length
if (chunklen)
{
//Loop for chunklen bytes of data
len=0;
do
{
if (chunklen > sizeof(Buffer))
{
//Recieve sizeof(Buffer) bytes of binary data
len = RecieveHTTPData(s, Buffer, sizeof(Buffer), 0x00);
chunklen -= len;
}
else
{
//Recieve chunklen bytes of binary data
len = RecieveHTTPData(s, Buffer, chunklen, 0x00);
chunklen = 0;
}

//write the data to disk
fwrite(Buffer, len, 1, fileptr);

}while(chunklen);

}
else
{
//chunklen is zero, signifying end of transmission
Done=1;
}
}while(!Done);

//all done, cleanup as we leave
printf("\
Done, command output written to [result.dat].\
");
fclose(fileptr);
closesocket(s);
}


// ************************************************************ **********************************
// A helper function for dealing with HTTP chuncked data
// We have to do this when using WinSock, because WinSock sockets are NOT streams
// in a Unix system a socket is also a file descriptor allowing use of stream I/O
// ************************************************************ **********************************
int RecieveHTTPData(SOCKET s, char *Buffer, int Len, char Terminator)
{
static int PacketLen=0;
static int SrcIndex=0;
static int DstIndex=0;
static char PacketBuffer[1500];
char c;

//Destination is zero lenght each call
DstIndex=0;

if (Terminator)
{
//Were retrieving terminated data
do
{
//Retrieve HTTP data as required into PacketBuffer[]
if (SrcIndex == PacketLen)
{
PacketLen = recv(s, PacketBuffer, sizeof(PacketBuffer), 0);
SrcIndex = 0;
}

//Copy a byte from PacketBuffer[SrcIndex] to Buffer[DstIndex]
c = Buffer[DstIndex++] = PacketBuffer[SrcIndex++];
Buffer[DstIndex]=0x00;

}while(c != Terminator);
}
else
{
//Were retrieving binary data
do
{
//Retrieve HTTP data as required into PacketBuffer[]
if (SrcIndex == PacketLen)
{
PacketLen = recv(s, PacketBuffer, sizeof(PacketBuffer), 0);
SrcIndex = 0;
}

//Copy a byte from PacketBuffer[SrcIndex] to Buffer[DstIndex]
c = Buffer[DstIndex++] = PacketBuffer[SrcIndex++];

}while(DstIndex < Len);
}

//Return number of bytes coppied
return(DstIndex);
}
King Of Bayonne is offline  
post #23 of 499 Old 01-02-2002, 12:24 PM
AVS Special Member
 
toots's Avatar
 
Join Date: Aug 2000
Location: Bedford, NH USA
Posts: 4,084
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally posted by King Of Bayonne
SnapshotDir contains:

RemoteSnapshots
Snapshot-Local-1009926000

I have no idea what these are...
And thanks to you all who helped make this work!

I think I've mentioned this before, but if you ever yank the drive on the replay and dump all the files out of the system partition, you will find (from memory, so excuse me if I get the precise location wrong) /sys2/etc/httpfs-export, which defines these three top level directories (Video, SnapshotDir, Photo). The SnapshotDir is some directory on the system partition, and while I didn't look too closely at its contents the last time I was there, I figured it to be some subset of the program guide, although I could certainly well be wrong about that.

Predictably, "Video" just points at the MPEG (big) partition, and "Photo" just points to the photo partition.

Everyone sucks more than everyone else
host of the Curmudgeon Message Boards
toots is offline  
post #24 of 499 Old 01-02-2002, 12:29 PM
Advanced Member
 
FlipFlop's Avatar
 
Join Date: Nov 2000
Location: Michigan
Posts: 944
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
A suggestion:

As a courtesy, could you attach the source code as an attachment instead of pasting into the message text. This keeps the thread from stretching out onto mutliple pages, and also will preserve the indenting in the file.
FlipFlop is offline  
post #25 of 499 Old 01-02-2002, 12:48 PM
Member
 
dfrumin's Avatar
 
Join Date: Aug 2001
Location: Seattle, WA, USA
Posts: 83
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Very cool. I spent a few days on this last week but then took a couple of days off.

I have a rudimentary web server working that serves up all but video. I'll try and finish it today and post it up. Some extra information...

Discovery:
Helps to change the serial number of the unit. I had it choke a couple of times and that helped. That also shows up in the uPnP publishing. Discovery also expires every 30mins, so I'm adding code to my server to send an "alive" SSDP message every 30mins.

Channel Guide:
The RTV will choke if we serve it the same channel guide twice (trust me, bad, required 777-zones.) The algorithm it follows is: it asks for serial_no=# and you either return it a fresh guide with a different file name or "up to date". My mini-server names the guide the way the replay expects it on the hard drive and checks that. So we have that piece.

Channel Guide format:
I haven't focused too much here cuz I want to serve video first, but this will become important. It carries an 808 bytes header, 624 bytes per replay channel, and 444 per show. I have most of the channel format decoded and will post that. Eventually I hope to write a channel guide generator.

Wacky enough, I ran into the basic auth piece to get the video but still ran into the access denied (80820004) error. I'll try again with the code posted to see what's up with that.

I'm going to try and integrate the learnings from the code posted into my server. Hopefully by EOD I can have a mini-server that serves a single show to a single RTV (threading and what not after that.)

thx.
dfrumin is offline  
post #26 of 499 Old 01-02-2002, 12:55 PM
AVS Special Member
 
toots's Avatar
 
Join Date: Aug 2000
Location: Bedford, NH USA
Posts: 4,084
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Ah, so the stuff in the SnapshotDir is just that: A snapshot of the program guide, with the Unix timestamp of when the snapshot of the guide was taken, encoded into the filename.

Right?

I mean, I'm just guessin' and all...

and thanks for straightening me out on what 80820004 means.

Everyone sucks more than everyone else
host of the Curmudgeon Message Boards
toots is offline  
post #27 of 499 Old 01-02-2002, 01:03 PM
Member
 
dfrumin's Avatar
 
Join Date: Aug 2001
Location: Seattle, WA, USA
Posts: 83
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Very cool. I spent a few days on this last week but then took a couple of days off.

I have a rudimentary web server working that serves up all but video. I'll try and finish it today and post it up. Some extra information...

Discovery:
Helps to change the serial number of the unit. I had it choke a couple of times and that helped. That also shows up in the uPnP publishing. Discovery also expires every 30mins, so I'm adding code to my server to send an "alive" SSDP message every 30mins.

Channel Guide:
The RTV will choke if we serve it the same channel guide twice (trust me, bad, required 777-zones.) The algorithm it follows is: it asks for serial_no=# and you either return it a fresh guide with a different file name or "up to date". My mini-server names the guide the way the replay expects it on the hard drive and checks that. So we have that piece.

Channel Guide format:
I haven't focused too much here cuz I want to serve video first, but this will become important. It carries an 808 bytes header, 624 bytes per replay channel, and 444 per show. I have most of the channel format decoded and will post that. Eventually I hope to write a channel guide generator.

Wacky enough, I ran into the basic auth piece to get the video but still ran into the access denied (80820004) error. I'll try again with the code posted to see what's up with that.

I'm going to try and integrate the learnings from the code posted into my server. Hopefully by EOD I can have a mini-server that serves a single show to a single RTV (threading and what not after that.)

thx.
dfrumin is offline  
post #28 of 499 Old 01-02-2002, 01:13 PM
Senior Member
 
karog's Avatar
 
Join Date: Aug 2000
Posts: 466
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally posted by King Of Bayonne
mlinehan, your Auth data works fine with my box.
Yes, that auth data is the same both my 3060 and SS use to get nightly guide data from the mothership.

The data is just a base64 encoding of username:password where username in this case is RNSBasic and password is some letters, digits, and symbols. I won't provide that so as not to piss off any ReplayGuys but it is easily gotten from the data since base64 is easily reversible.

base64 is encryption only from the casual glance. Its real purpose is to turn byte streams into printable character (A-Za-z0-9+/=) streams.

Also, where you said secure connection, you probably meant authorized connection, no? Secure implies something like HTTPS (SSL).
karog is offline  
post #29 of 499 Old 01-02-2002, 02:04 PM
Member
 
King Of Bayonne's Avatar
 
Join Date: Dec 2001
Location: Beautiful crime-free Albany, NY
Posts: 28
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally posted by FlipFlop
As a courtesy, could you attach the source code as an attachment instead of pasting into the message text. This keeps the thread from stretching out onto mutliple pages, and also will preserve the indenting in the file.
Guilty as charged. Normally, I'd attach the file or put it somewhere Internet-accessible, but I was so damned excited... :)


KOB
King Of Bayonne is offline  
post #30 of 499 Old 01-02-2002, 02:05 PM
Member
 
King Of Bayonne's Avatar
 
Join Date: Dec 2001
Location: Beautiful crime-free Albany, NY
Posts: 28
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally posted by karog

Also, where you said secure connection, you probably meant authorized connection, no? Secure implies something like HTTPS (SSL).
Correct... I mean "authorized."


KOB
King Of Bayonne is offline  
Closed Thread ReplayTV & Showstopper PVRs



Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off