Panasonic DMR hard drive data recovery - It CAN be done!! - Page 3 - AVS Forum
Forum Jump: 
Reply
Thread Tools
post #61 of 168 Old 04-26-2010, 10:02 AM
Newbie
 
0x1BBE898's Avatar
 
Join Date: Apr 2010
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Wow, this was a fast response.
Quote:
Originally Posted by i86time View Post

I'm definitely interested. Three potential problems:
1) I'm running WinXP (but I know how to run a knoppix disk)

As far as I can see the script should work also on Windows (Python interpreter required of course).

Quote:
Originally Posted by i86time View Post

2) My data is now stored as an image file

This should make the process easier.

Quote:
Originally Posted by i86time View Post

3) The data came from a HDD that required a format. Panny HDD formats are 'simple' in that they only overwrite the TOC (or whatever the correct linux term is) - the actual mpeg data is still there, just no direct pointers to what is what.

Yes, I definitly found deleted recordings. But the script doesn't touch the TOC of the filesystem. Reverse engineering the filesystem is a bit too hard for me It looks for mpeg-headers and uses the timecode to split the chunks.
If some parts are too small (size below an adjustable value), the script will ignore them. I used a value of 50 MiB.

Quote:
Originally Posted by i86time View Post

Let me know if you think it'd still be OK to use and what I'd need.

There should be no problem at all. The last feature of the script (dump chunks to file) is not ready yet. I've also found another bug in the code. Give me a few days.

Nice to hear that somebody has interest.
0x1BBE898 is offline  
Sponsored Links
Advertisement
 
post #62 of 168 Old 04-26-2010, 03:35 PM
Newbie
 
bameije's Avatar
 
Join Date: Apr 2010
Posts: 3
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
@0x1BBE898
wow, this is amazing. 2 days ago I found this thread. Today I sit in front of my computer to actually start creating the very script you're talking about. Just with luck I saw your post that you just posted yesterday. After 1 year of inactivity!

This said, I'm *very* interested in your script... I am running Windows, but I have a VirtualBox running Ubuntu, that should do. I've got an image of my HDD. It's split into 1GB chucks so there might be some work there.

Another drawback is that I'm not really familiar with Python, I was planning to write C++... but I'll jump into it right now.

Hope you find time to do the 'dump chunks to file' bit, although I understood that you had already recovered your drive?
bameije is offline  
post #63 of 168 Old 04-27-2010, 06:43 AM
Newbie
 
0x1BBE898's Avatar
 
Join Date: Apr 2010
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by bameije View Post

I've got an image of my HDD. It's split into 1GB chucks so there might be some work there.

The current script can't handle this situation. But if I have the time, i will try to solve this. Shouldn't be too hard...

Quote:
Originally Posted by bameije View Post

Another drawback is that I'm not really familiar with Python, I was planning to write C++... but I'll jump into it right now.

The first prototype I've written in C. But Python has much more power and there is no need to reinvent the wheel The Python script is only insignificant slower than the C program.

Quote:
Originally Posted by bameije View Post

Hope you find time to do the 'dump chunks to file' bit, although I understood that you had already recovered your drive?

You're right. The export feature was only a fast hack to quickly review the result.
0x1BBE898 is offline  
post #64 of 168 Old 04-27-2010, 11:03 AM
Newbie
 
0x1BBE898's Avatar
 
Join Date: Apr 2010
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Have a look at: http://sites.google.com/site/haliner/dvr-recover

A short introduction:

If you think your hdd image is important, first make a backup There is absolutely no warranty.

Download and extract the script (or checkout the git-repo) and open a shell:
Code:
$ cd /path/to/script
$ python dvr-recover.py usage
This should print the usage information.

Now you can create a sample configuration file:
Code:
$ python dvr-recover.py sample_settings
Edit the settings file. (See usage for more information. The most import keys are 'hdd-filename', 'chunk-filename' and 'export-dir'.)

Create chunk list: (May take a long time! I used only a very small part of the whole hdd.)
If you abort this process, all gathered information are lost! The script saves the information only at the very end.
Code:
$ python dvr-recover.py create
[ 46.8%] 471420/1006929 blocks (15713.9 bl/s; 30.7 MiB/s): 6 chunks
[ 93.5%] 941768/1006929 blocks (15678.2 bl/s; 30.6 MiB/s): 6 chunks

Finished.
Read 1006929 of 1006929 blocks.
Found 7 chunks.
Took 64.08 seconds.
Average speed was 15714.0 blocks/s (30.7 MiB/s).
Show chunk information:
Code:
$ python dvr-recover.py show
Sort chunks:
Code:
$ python dvr-recover.py sort
Show chunk information again:
Code:
$ python dvr-recover.py show
--+--------------+--------------+--------------+--------------+-------------
  |  Block Start |   Block Size |  Clock Start |    Clock End | Concatenate
--+--------------+--------------+--------------+--------------+-------------
0 |       391379 |       615548 |    250031969 |    402836926 |    False
1 |       223188 |       168189 |    401478251 |    484558534 |    False
2 |       107989 |       115197 |    546892241 |    592303341 |    False
3 |        80992 |        26995 |   1085209725 |   1100156779 |    False
4 |        53995 |        26995 |   1100157071 |   1115282776 |     True
5 |        26998 |        26995 |   1115283069 |   1129877061 |     True
6 |            0 |        26996 |   1129877208 |   1144930337 |     True
Extract fourth chunk: (To /chunk_####.mpg)
Code:
$ python dvr-recover.py export 3
Current chunk: #3
Chunk start: 80992
Chunk size:  26995
 0.95s (28436.33 blocks/s; 55.54 MiB/s).

Concatenate chunk: #4
[...]

Concatenate chunk: #6
Chunk start: 0
Chunk size:  26996
 3.22s (8388.86 blocks/s; 16.38 MiB/s).
Extract all chunks:
Code:
$ python dvr-recover.py export
Current chunk: #0
Chunk start: 391379
Chunk size:  615548
 42.22s (14579.92 blocks/s; 28.48 MiB/s).
[...]
Please let me know, if the script is working or not (and if you like it or not). If you have any additional questions, feel free to ask me!

This was only a very very short intro, hope it’s comprehensible. But now I need a break

Have fun!
0x1BBE898 is offline  
post #65 of 168 Old 04-27-2010, 10:19 PM
Member
 
i86time's Avatar
 
Join Date: Jul 2003
Posts: 120
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 10
Thanks for this. Give me a few days or so to practice with Python, then I'll tackle this project and report back with results.
i86time is offline  
post #66 of 168 Old 05-07-2010, 03:52 PM
Newbie
 
bameije's Avatar
 
Join Date: Apr 2010
Posts: 3
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I must say 0x1BBE898, that's a terrific job you've done there!

As I said, my problem is I've got my 160GB split up in smaller files. They're actually 4GB (not 1GB as I stated before) or around that. They're always split on block borders (so each file starts with the mpeg code 00 00 01 BA).

So I guess I need to change your script to do the following:
-with every chunk: save an extra parameter that says in which file the chunk was found.
-Now I've got the advantage that I can run the script on the smaller files, and append the chunk data to the chunk file, rather then overwriting. So the script doesn't have to process the whole 160GB before the chunk data is written.
-Sorting the chunks doesn't change.
-exporting the chunks: always check in which file the chunk was found, then open that file and export from it.

Am I correct to say that these are the only changes that need to be made?

Thanks for the great script.
bameije is offline  
post #67 of 168 Old 05-08-2010, 02:47 AM
Newbie
 
0x1BBE898's Avatar
 
Join Date: Apr 2010
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I guess you're right. But I have another idea... There should be only a few changes necessary:

Instead of one input file we will use a list of those. The settings file will simply contain more than one input-file-entry. For every file in the list, the script runs os.stat(path) and so it's easy to determine the size of each file.

Only one more simple change is necessary: The offset must be calculated with all previous file sizes in mind. But that's quite easy

I think this solution also works with files which aren't split on block borders.

Maybe till next week I have a working script for split files. Shouldn't take too much time.
0x1BBE898 is offline  
post #68 of 168 Old 05-09-2010, 07:47 AM
Newbie
 
0x1BBE898's Avatar
 
Join Date: Apr 2010
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
It was much more easier than I had expected. Only a small amount of time was necessary to implement handling of multiple input streams.

http://sites.google.com/site/haliner/dvr-recover

Additional info: The resulting mpeg streams can easily edited with tools like Avidemux (http://avidemux.sourceforge.net/). I removed unnecessary content at the beginning / end.
0x1BBE898 is offline  
post #69 of 168 Old 05-10-2010, 12:36 AM
Member
 
i86time's Avatar
 
Join Date: Jul 2003
Posts: 120
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 10
I'm afraid I'm lost already...
So I've installed Python 2.6 (ActiveState ActivePython). I've set Python in my path... When I use the GUI and Dos Shell, and run "dvr-recover.py sample_settings" I get the following error:
Code:
Traceback (most recent call last):
  File "dvr-recover.py", line 779, in 
    Main().run()
  File "dvr-recover.py", line 771, in run
    self.load_settings()
  File "dvr-recover.py", line 388, in load_settings
    f = open(self.settings_filename, 'r')
IOError: [Errno 2] No such file or directory: 'dvr-recover.conf'
So, is it expecting that I've already created a .conf file? Even when I create a dummy dvr-recover.conf with the following settings:
Code:
hdd-file=[L:\\DMREH75recover2-img\\DMREH75V.img]
chunk-file=[L:\\chunk]
export-dir=[L:\
ecover]
blocksize=[2048]
min-chunk-size=[25600]
max-create-gap=[90000]
max-sort-gap=[90000]
I get the following error:
Code:
Traceback (most recent call last):
  File "dvr-recover.py", line 624, in 
    Main().run()
  File "dvr-recover.py", line 616, in run
    self.load_settings()
  File "dvr-recover.py", line 251, in load_settings
    self.blocksize = int(value)
ValueError: invalid literal for int() with base 10: '[2048]'
So, I've got to be doing something wrong. Any help?
i86time is offline  
post #70 of 168 Old 05-10-2010, 09:35 AM
Newbie
 
0x1BBE898's Avatar
 
Join Date: Apr 2010
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by i86time View Post

I'm afraid I'm lost already...
So I've installed Python 2.6 (ActiveState ActivePython). I've set Python in my path...

I have never heard about ActivePython. I recommend using the plain CPython implementation (2.5 or never, but not 3.x): http://python.org/download/ But that's not the point for the error messages.

Quote:
Originally Posted by i86time View Post

When I use the GUI and Dos Shell, and run "dvr-recover.py sample_settings" I get the following error:
Code:
Traceback (most recent call last):
  File "dvr-recover.py", line 779, in 
    Main().run()
  File "dvr-recover.py", line 771, in run
    self.load_settings()
  File "dvr-recover.py", line 388, in load_settings
    f = open(self.settings_filename, 'r')
IOError: [Errno 2] No such file or directory: 'dvr-recover.conf'
So, is it expecting that I've already created a .conf file?

Aaargh. That's a bug. The script tries to read the settings file at startup, but this is definitely wrong for the sample_settings-parameter.

Quote:
Originally Posted by i86time View Post

Even when I create a dummy dvr-recover.conf with the following settings:
Code:
hdd-file=[L:\\DMREH75recover2-img\\DMREH75V.img]
chunk-file=[L:\\chunk]
export-dir=[L:\
ecover]
blocksize=[2048]
min-chunk-size=[25600]
max-create-gap=[90000]
max-sort-gap=[90000]
[...]

So, I've got to be doing something wrong. Any help?

The square brackets are wrong. The documentaton is somewhat imprecise in this context. Should look like this:
Code:
hdd-file=L:\\DMREH75recover2-img\\DMREH75V.img
chunk-file=L:\\chunk
//edit: See http://sites.google.com/site/haliner/dvr-recover
I hope the square brackets in the usage message are now a bit clearer.
0x1BBE898 is offline  
post #71 of 168 Old 05-10-2010, 11:54 AM
Member
 
i86time's Avatar
 
Join Date: Jul 2003
Posts: 120
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 10
Thanks for your quick reply. I uninstalled ActivePython and installed CPython 2.6.5 . I still must have something wrong though. First, I can't "import" the dvr.recover.py script into the Python sys.path or whatever, I need to go to a DOS shell, change to the directory, then run the script from there, so I'm sure I've overlooked something. But I did edit my .conf file and removed all brackets. When I run "dvr-recover samples_settings", no errors are noted, and the command line comes up again. So I then run "dvr-recover create" and I get the following:

Code:
C:\\Python26\\Lib\\site-packages\\dvr-recover-0.2>dvr-recover create
Traceback (most recent call last):
  File "C:\\Python26\\Lib\\site-packages\\dvr-recover-0.2\\dvr-recover.py", line 779,
 in 
    Main().run()
  File "C:\\Python26\\Lib\\site-packages\\dvr-recover-0.2\\dvr-recover.py", line 773,
 in run
    func()
  File "C:\\Python26\\Lib\\site-packages\\dvr-recover-0.2\\dvr-recover.py", line 621,
 in create
    reader = FileReader(self.input_filenames)
  File "C:\\Python26\\Lib\\site-packages\\dvr-recover-0.2\\dvr-recover.py", line 245,
 in __init__
    part.size = os.stat(part.filename).st_size
WindowsError: [Error 3] The system cannot find the path specified: ''
Any ideas what is happening? All directories/files in my .conf file exist.
i86time is offline  
post #72 of 168 Old 05-10-2010, 12:16 PM
Newbie
 
0x1BBE898's Avatar
 
Join Date: Apr 2010
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by i86time View Post

Thanks for your quick reply. I uninstalled ActivePython and installed CPython 2.6.5 . I still must have something wrong though. First, I can't "import" the dvr.recover.py script into the Python sys.path or whatever, I need to go to a DOS shell, change to the directory, then run the script from there, so I'm sure I've overlooked something.

No, that's exactly the way to go. It looks a bit uncomfortable first, but I think every Linux (well Unix, ...) user get's addicted to it
Importing is much more complicated.

Quote:
Originally Posted by i86time View Post

But I did edit my .conf file and removed all brackets. When I run "dvr-recover samples_settings", no errors are noted, and the command line comes up again. So I then run "dvr-recover create" and I get the following:
[...]
Any ideas what is happening?

You should run the script with the parameter sample_settings and after that edit the settings file.

If you edit the file first and run sample_settings afterwards, the script will overwrite your changes.

@bameije: Have forgotten to mention it, this small script helps to build a settings file for many input files:
Code:
#!/usr/bin/env python
# -*- coding: utf-8 -*-

f = open('hdd-files.txt', 'w')
for i in xrange(0,50):
    print >>f, 'hdd-file=path-to-hdd-files/part_%03i.img' % i
f.close()
It creates a file called hdd-files.text containing the string hdd-file=... multiple times. %03i will be substituted with the number in variable i. The resulting string has at least 3 digits. The 0 in front of the length-modifier will pad the value with zeros:
Code:
hdd-filename=path-to-hdd-files/part_000.img   
hdd-filename=path-to-hdd-files/part_001.img   
hdd-filename=path-to-hdd-files/part_002.img
0x1BBE898 is offline  
post #73 of 168 Old 05-10-2010, 05:44 PM
Member
 
i86time's Avatar
 
Join Date: Jul 2003
Posts: 120
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 10
Quote:
Originally Posted by 0x1BBE898 View Post

No, that’s exactly the way to go. It looks a bit uncomfortable first, but I think every Linux (well Unix, ...) user get’s addicted to it
Importing is much more complicated.

You should run the script with the parameter „sample_settings“ and after that edit the settings file.

If you edit the file first and run „sample_settings“ afterwards, the script will overwrite your changes.

Hello,
Thanks again for that speedy response. You are correct, after I ran w/ the sample_settings switch, it overwrote my old .conf file and erased my directories, which is why it errored out. However, aarrrggh! I got 'create' to work, and after it reached 100%, it gave me this:
Code:
Finished.
Read 39092004 of 39092004 blocks.
Found 150 chunks.
Took 15404.80 seconds.
Average speed was 2537.7 blocks/s (5.0 MiB/s).
Traceback (most recent call last):
  File "C:\\Python26\\Lib\\site-packages\\dvr-recover-0.2\\dvr-recover.py", line 779,
 in 
    Main().run()
  File "C:\\Python26\\Lib\\site-packages\\dvr-recover-0.2\\dvr-recover.py", line 773,
 in run
    func()
  File "C:\\Python26\\Lib\\site-packages\\dvr-recover-0.2\\dvr-recover.py", line 626,
 in create
    self.save_chunk_list()
  File "C:\\Python26\\Lib\\site-packages\\dvr-recover-0.2\\dvr-recover.py", line 376,
 in save_chunk_list
    f = open(self.chunk_filename, 'w')
IOError: [Errno 13] Permission denied: 'L:\\\\chunk'
Notice the very last line, "L:\\\\chunk" I checked and double checked my .conf file and the entry is "L:\\chunk" (one forward slash only).

EDIT:
'test-settings' showed this:
Code:
C:\\Python26\\Lib\\site-packages\\dvr-recover-0.2>dvr-recover test_settings
input-file: ['L:\\\\DMREH75recover2-img\\\\DMREH75V.img']
chunk-file: L:\\chunk
export-dir: L:\
ecover
blocksize: 2048
min-chunk-size: 25600
max-create-gap: 90000
max-sort-gap: 90000
So it added an extra slash to my input file directory. But seeing as it found it anyway, seems like it should work, right? Weird that it has brackets around the input file (I did not enter those), but not on the chunk or export directory.

Oh, and is there a way that I can select just a portion of the .img to run this on so that I can check to make sure it works before doing the whole thing?
i86time is offline  
post #74 of 168 Old 05-11-2010, 02:22 AM
Newbie
 
bameije's Avatar
 
Join Date: Apr 2010
Posts: 3
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by 0x1BBE898 View Post

implement handling of multiple input streams.

Thanks! It works like a charm... I only tested it on 8 x 1GB test chunks. Can only unleach it on my real 40 x 4GB data in 2 weeks, but I'm sure it'll do just fine.

Quote:
Originally Posted by i86time View Post

Oh, and is there a way that I can select just a portion of the .img to run this on so that I can check to make sure it works before doing the whole thing?

Yes. One of many ways to do it: just open your .img in HxD and save only the first 1GB in another file (eg .img.part). Use this file in your dvr-recover.conf file. The script should find a few chunks in this partial file as well...
Good luck!
bameije is offline  
post #75 of 168 Old 05-11-2010, 10:19 AM
Newbie
 
0x1BBE898's Avatar
 
Join Date: Apr 2010
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by i86time View Post

I got 'create' to work, and after it reached 100%, it gave me this: [...]
Notice the very last line, "L:\\\\chunk" I checked and double checked my .conf file and the entry is "L:\\chunk" (one forward slash only).

That’s a pity. After this long time an error...

In Python “\\” is an escape character. So internally a single “\\” in real world will be handled as something like “\\\\”. That’s not a problem. So if you specified the path in the settings file correctly, you’re fine. (And even if, Windows is also able to handle “\\\\” in paths correctly.)

There must be another problem. Probably your user has no permissions to write to L:\\. You should check this.

The script should try to open the chunk file before scanning the input files. This way the script crashes before and not after the time-consuming process. A bit quick and dirty, but better than nothing at all. (*)

It would be fantastic if the script could save it’s current state to a file. So a later resuming without losing information would be possible. Never implemented something like that, but it’s a nice playground. Some ideas are growing in my head...

Quote:
Originally Posted by i86time View Post

Weird that it has brackets around the input file (I did not enter those), but not on the chunk or export directory.

Also a minor cosmetic mistake I order Python to print the list of all filenames. And Python is using square brackets for lists. (*)

Quote:
Originally Posted by i86time View Post

Oh, and is there a way that I can select just a portion of the .img to run this on so that I can check to make sure it works before doing the whole thing?

No, but see bameije’s post. (You can get HxD from here.)

@bameije: Ok, great

(*) = Fixed in current release, see http://sites.google.com/site/haliner/dvr-recover

//edit: Tested the current release on Windows. Everything went smoothly.
0x1BBE898 is offline  
post #76 of 168 Old 05-11-2010, 11:21 AM
Member
 
i86time's Avatar
 
Join Date: Jul 2003
Posts: 120
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 10
OK, I don't know how it happened, but I got it to work. I ended up retyping in my directory values... I think I may have had an extra space after my chunk file value in sample_settings. I then followed the remaining steps using a 1GB portion of my image and it found and exported a part of a much larger file from the HDD, so I've got it working. Now I just need to run it again for the 4 hours and hope for no other problems. I'll report back when finished. Thanks again for the help from both of you!
i86time is offline  
post #77 of 168 Old 05-11-2010, 07:20 PM
Member
 
i86time's Avatar
 
Join Date: Jul 2003
Posts: 120
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 10
Well, that worked quite well. Thank you muchly for such a great tool that's an insane timesaver. I was able to recover 45 files, about 67 GB total. I can't remember how full the drive was, but that seems to be everything. There was only a couple small hitches. One .mpg is nearly unplayable. It starts, but 10 secs in or so and it causes the player to shutdown. I've tried it on WMP, WMP Classic, Player, PowerDVD, etc. (will try it on VLC later). It won't even play on VideoReDo, which will play and fix pretty much any .mpg frame/header error, so I don't hold much hope on recovery. The other problem I had was that the program concatenated 2 different files. It combined the beginning of one file with the end of another. Then it saved the end of the first file and the beginning of the other separately from chunks later down the list. Other than that, if worked perfectly! It even had the bits intact that I originally trimmed from the recordings.

One question. The smallest file it found was ~ 52 MB (0:50 seconds or so), as per the settings file. I don't recall exactly, but I may have had snippets smaller than that. If so, is there any way to re-run this program with the smaller chunk size to find ONLY files smaller than that 50MB limit and save only those to a chunk file to export again?

Thanks again for such a great tool!
i86time is offline  
post #78 of 168 Old 05-19-2010, 09:31 AM
Newbie
 
smith321's Avatar
 
Join Date: May 2010
Posts: 1
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Panasonic have proved themselves leaders in the field of Freeview DVD Recorders with integral hard disk technology. Their latest products include the DMREX88, with it's massive 400GB disk and the DMREX768, a replacement of the popular DMREX77 DVD recorder with a 160GB HDD.

In this review we take a look at the DMR-EX77 and DMR-EX87 DVD recorders, both of which incorporate a Freeview compatible DVB-T tuner and large capacity , together with providing an HDMI interface with up-scaling to full HD (1080p / 1080i). In keeping with other Panasonic DVD recorder models, both offer exceptional picture quality when connected to an HD ready flat screen TV.
smith321 is offline  
post #79 of 168 Old 05-19-2010, 10:14 AM
AVS Special Member
 
Westly-C's Avatar
 
Join Date: Jun 2004
Location: Techno World
Posts: 3,076
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 83 Post(s)
Liked: 74
^^You do know that flaunting Panasonic's non US based models can cause grown men to weep, don't ya?

Dazed and confused over high tech.

Sigh...Concrap. The Internet Overlord Cometh
They're not com-tastic!
Westly-C is online now  
post #80 of 168 Old 06-19-2010, 02:35 PM
Newbie
 
0x1BBE898's Avatar
 
Join Date: Apr 2010
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I released dvr-recover 0.5: http://sites.google.com/site/haliner/dvr-recover

The script uses a sqlite database now. It is now possible to interrupt the process of gathering all the chunk info. The script will continue the process automatically.

You can find documentation on the project page and in the source tarball or zipfile. Ask whenever you have a question or something is unclear.

I think I need not to say that feedback is very welcome

Regards,
Stefan
0x1BBE898 is offline  
post #81 of 168 Old 07-06-2010, 07:20 PM
Newbie
 
gsouth's Avatar
 
Join Date: Oct 2004
Posts: 7
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I have a DMR-XW300 that the DVD drive has packed up on - been using the scripts to try and extract some video and getting nowhere fast. REtrieved 4 video's but then the script crashes.

In desperation I downloaded the drive using dd to my mac (all 250GB of it) into a disk image file, and ran ISOBuster over it - its' found just about every video file and I'm in the process of extracting them now....
gsouth is offline  
post #82 of 168 Old 07-07-2010, 10:46 AM
Newbie
 
0x1BBE898's Avatar
 
Join Date: Apr 2010
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by gsouth View Post

In desperation I downloaded the drive using dd to my mac (all 250GB of it) into a disk image file, and ran ISOBuster over it - its' found just about every video file and I'm in the process of extracting them now....

Interesting. I've forgotten to check if IsoBuster can open the image. Unfortunately I removed the image of the hdd, because it wasted my disk space. So I can't test it now.

Maybe Panasonic uses some kind of UDF as file system. According to Wikipedia a block size of 2048 byte is typical for the UDF file system. In the most common case UDF is used as a read-only file system on CDs and DVDs. But it is also designed to be writable. A UDF like file system would explain why IsoBuster is able to open the image.

I would be interested in a screenshot of IsoBuster showing the content of the image (filenames and directory structure). Anyone who can provide one?

Thanks in advance
0x1BBE898 is offline  
post #83 of 168 Old 07-07-2010, 04:29 PM
Member
 
i86time's Avatar
 
Join Date: Jul 2003
Posts: 120
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 10
IsoBuster can open the image (like some other image programs). But also like those other programs, it just finds a series of mpegs (it labels them as .VOB's) and some very large .IFO files that must also be mpeg video. On my image, aside from 5 small files, it separated the 80 GB image into 80+ ~1 GB files. I don't have a license, so I couldn't test it beyond that, but it seems it's the same as all the others; it'll copy over all the video files, but it won't concatenate them.

I burn UDF DVD's when I have files larger than 2GB. I don't know why recorders would need to create a file that large, as it appears they create 1GB chunks. Who knows?
i86time is offline  
post #84 of 168 Old 07-10-2010, 02:48 AM
Newbie
 
gsouth's Avatar
 
Join Date: Oct 2004
Posts: 7
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
its' definitely using UDF of some description for the storage - it kind of makes sense - then it doesn't matter if you are writing the file to DVD-RAM or HDD it would be treated the same.
However I also think they are using some other file system for managing the title list -

ISOBuster could open the image file, but only after you force it to as it doesnt' think it's a UDF file at the start (there's a LOT of padding at the beginning of the image as well as some references to another file system type) and definitely recovered a lot of files.
However I found it corrupted the output files with various streams overwriting each other ans starting half way into each other.

I have another patched version of the python scripts that has allowed me to extract many of the files, however its' still missing some key files so my journey to recover this thing still continues.
gsouth is offline  
post #85 of 168 Old 07-11-2010, 04:42 AM
Newbie
 
0x1BBE898's Avatar
 
Join Date: Apr 2010
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I really don’t know if the recorder is using UDF or any similiar file system. I only come accross the fact that the block size of UDF is 2048 byte (like the one of the hdd) by chance. I would make sense, because the recorder doesn’t have to convert between file systems and file types.

As far as my script is useful for someone I’ll try to maintain it, even if IsoBuster would open the image accordingly.

The firmware of the recorder is also located at the beginning of the disk. So it’s really suprising that IsoBuster can open the image.

I fixed a bug claiming that there are multiple references to a chunk. You can find the new release (the one mentioned by gsouth + a minor source code cleanup) like all other releases on the homepage: http://sites.google.com/site/haliner/dvr-recover

Personally I think that the default value for “min_chunk_size” is quite too high. So if someone can prove that a lower value is better, I’ll change the default. See section “Tuning the parameters” on the project page.
0x1BBE898 is offline  
post #86 of 168 Old 07-11-2010, 11:08 AM
Member
 
i86time's Avatar
 
Join Date: Jul 2003
Posts: 120
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 10
Quote:
Originally Posted by 0x1BBE898 View Post

As far as my script is useful for someone I'll try to maintain it, even if IsoBuster would open the image accordingly.

Definitely do not stop working on/hosting this program. As I stated, most ISO/data recovery programs will find the mpeg data, but that's only half the desired result. None of the ones I have tried will even remotely reassemble the data into the original watchable file. To do what your program does, one would need an ISO/recovery program, an editing program (like VideoReDo) AND countless hours. But if someone wants to reinvent the wheel, I guess they could try.


Quote:


Personally I think that the default value for min_chunk_size is quite too high. So if someone can prove that a lower value is better, I'll change the default. See section Tuning the parameters on the project page.

Well, for someone like me, it may be a bit large. 50MB translates to ~ 50 secs @ XP, 100 secs @ SP and so on. I recorded little snippets of kids programs off Nick JR for my daughter, so I'm guessing some of those were missed. Also, perhaps if one has a drive that at one point had been nearly full, then when recording a long program that doesn't have all the required contiguous space, it could potentially place < 50 MB portions of the recording throughout the drive, which could be skipped. I have no idea what a good new minimum chunk size would be though.
i86time is offline  
post #87 of 168 Old 07-11-2010, 11:46 AM
Newbie
 
0x1BBE898's Avatar
 
Join Date: Apr 2010
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
As I stated above, it's not my intention to give up development. The topic is also quite interesting for me and hosting doesn't cost me anything.

The 50MB limit is taken over from an older version of the program (which worked differently).

I have another idea: Why limit the size while gathering the data? The script should analyze the disk completely and save everything. A limit while exporting whole recordings makes sense, because very short recordings are unusual. But the limit shouldn't be too high either, because it can be just a part and the program hasn't detected this correctly.

I expect that the results will be much better and more data can be restored.

//edit:
For those who wants to experiment:
Code:
python dvr-recover.py setup min_chunk_size 0
After that go through the whole process again (create, sort, export).

No chunks will be ignored after running the command above. Hopefully more files can be recovered.

I'll implement my idea in some time. But setting min_chunk_size to 0 is almost identical. (Only the small files are not ignored during the export.)
0x1BBE898 is offline  
post #88 of 168 Old 07-11-2010, 06:22 PM
Newbie
 
gsouth's Avatar
 
Join Date: Oct 2004
Posts: 7
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Hi,
ISOBuster can read teh file as it simply scans through it looking for UFD fragments and saves them - it works ok however as it has non knowledge of the underlying organisation I hve found it corrupts more files thatn it recovers.

I tried with min_chunk_size at 1000 last night and did get more files back but still am missing around 20-30%.

Going to look at the file structure more later this week - if I can extract an intact UFD file form the overall disk, put it back onto a blank drive then I should be able to mount it and read it......
gsouth is offline  
post #89 of 168 Old 08-14-2010, 05:49 PM
Newbie
 
DavidandSavi's Avatar
 
Join Date: Aug 2010
Posts: 1
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
All,

I have been watching this thread with must interest. The DVD-Drive on my Panny (DVR-EH55) died a while ago (and before people suggest cleaning etc..., done all that). The HDD is fine, I can watch the programs etc etc, just can't burn them to DVD, and there is stuff I want/need to keep (e.g copys from VHS tapes)

The quote for repair was AUS$250, which I figured wasn't worth it (thats half the cost of the latest twin-HD DVD/HDD recorder). Well, being an IT person, I figured, there HAS to be a way of pulling the data from the hard drive. Also, the HDD was just about full with many-many programs (my wife had done a lot of recording, and cutting out 3-5 min segments from various shows for me to watch).

Anyway, the picture I want to paint is that I want to be sure that I get ALL of the data off the HDD.

Following along the lines of what some others here have done, I have started examining the HDD via a hexeditor (I use Winhex) in an attempt to try to decipher the file system being used (There has to be some way that the Panasonic can piece together the fragments for a given show, and to provide a listing of all recorded programs etc etc)

So anyway, I did some scanning and started getting hits on programs I knew I recorded. These hits looked like Meta-data - Name of the program, date recorded, ...

Well, one of these strings "dvdvrx", kept coming up. I thought, oooh, maybe this is attributes (in a Unix way) for the files. So I did as anyone does these days, I googled the string.

Immediate success. I went to this page - freepatentsonline dot com Slash EP1091577 html. I can't post urls, as am a newbie to this forum, so hopefully you can work outthe link.

This is a Patent from MATSUSHITA (which is also Panasonic). I have downloaded the attached PDF and am currently reading it. All of a sudden the data in HDD is coming to life :-)

I still need to do a lot of investigation, but wanted/needed to share this information, in the hope that Stefan's dvd-recover could be tweaked to pass through the file directory. I don't know python so not really up to offer coding suggestions for it.

REgards

David
DavidandSavi is offline  
post #90 of 168 Old 09-20-2010, 09:50 AM
Newbie
 
im2020's Avatar
 
Join Date: Jan 2006
Posts: 8
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Hello Guys

Set out below is a copy of a post I submitted last July. It seems there were one or two requests for info and/or code around Christmas time but I wasn't alerted to them (think email notification was turned off) and have only just signed in for the first time this year.

So post again if anybody needs help on a situation similar to mine and I'll respond.

IM

============================

Copy of post on 12 July 2009

See my posts above in June 2009. I accidentally (re)formatted the 250GB hard drive on my Panasonic DMR-EX85EB thinking I was formatting a DVD-RAM.

The HDD contained 100 hours of treasures that I had not yet watched so after reading worley45 (wade)'s posts in this thread I was inspired to mount a project.

I took the HDD out of the Panny and connected it to my PC (with an IDE to USB adapter cable) and followed Wade's suggestions for using HxD (Hex editor) and HJSplit (file joiner/splitter) to retrieve my treasures and then burn them to DVD. I created a few tools to streamline the process and ended up with 25 DVDs containing all my treasures.

In the meantime, so that I was able to continue using my Panasonic DMR-EX85EB, I bought a replacement Western Digital Caviar WD2500BB 250GB 7200rpm IDE/PATA for £30. Before fitting this into my Panny I copied onto it the first 320KB from the old HDD (Panasonic's proprietary formatting code) and bingo on switching the Panny back on I was welcomed with the information that I had 111 hours of Standard Play recordable capacity.

So my message is that almost anything is possible.

If you want more details of anything described above or if you want a copy of the code to format a replacement 250 KB HDD post here and I'll respond.

im2020
im2020 is offline  
Reply DVD Recorders (Standard Def)



Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off