[Vlad Theimpaler]

I have some experiences to share regarding the task of copying very large files (>4GB, i.e. raw HDTV captures from HD Homerun) using Windows 7 x64.


I copy recordings of interest onto a NTFS-formatted 3.5" internal hard disk drive that I place into one of those eSATA docking stations for 3.5"/2.5" internal HDDs or SSDs (e.g. the Thermaltake BlacX). In a typical copy session, I transfer around 40 to 50 raw .tp files -- roughly 250GB to 300GB of total data -- from my Windows 7 x64 HTPC/DVR to a 2TB HDD inserted into a docking station and attached to the HTPC/DVR via an eSATA cable.


I copy the files using Windows 7's copy and paste function in Windows Explorer. But something I've noticed with this method is that right after the copy progress bar reaches 100% and the dialog box disappears (indicating the completion of the file transfer), data is still being transferred for several more minutes, despite the clear indication from Windows that the process was completed.


I found this out because of several instances in which I removed the external drive immediately after the Windows UI indicated the process finished... only to find that the last file to be transferred had been corrupted so badly that it could not be opened with either VLC or VideoReDo.


When I figured out that I needed to wait a few minutes after Windows said the transfer was "done", I never had any issues afterwards.


Anyway, while I was trying to research this phenomenon I learned about 3rd party tools like Teracopy and Robocopy, and while googling for details on those applications, I stumbled upon a pretty large thread on Microsoft's Social Technet forum that is titled "Windows 7 64-bit Corrupting (Altering) Large Files Copied to External NTFS Drives". In that thread discussion, a few posters claim that file transfer corruption is an issue when moving large sets of data from a Win7 machine to an external drive that is NTFS-formatted using Explorer copy/paste. The issue occurs regardless of the external interface, meaning both USB and eSATA connections have this problem.


Also mentioned in that thread is that general users will not likely notice the problem, particularly if their data consists of video files. This is because many playback systems have robust error correction algorithms, hence a few bit errors in a corrupted copy will not affect the playback of the video despite the fact that it does not match the original file bit-for-bit.


According to that thread, verification of the copy error is done using the CLI command fc /b file1 file2, where file1/file2 can be either original/copy or copy/original. An alternative check is the CLI command COMP. One post in the thread gave this as an example: comp test.mkv g:\test\test.mkv.


I was wondering if anyone here has observed any of these issues when copying large files to external NTFS-storage devices with the native Windows 7 copy/paste feature. Thanks for reading all this, and for any further discussion on the subject.

 

[gazebo]

I do this all the time -- copying gigabytes and terabytes of video files from internal disks and network disks to external disks in eSATA and USB3 docks. Never noticed a problem. The reason? I always use Windows' safely remove disk tool. You should never remove an external drive without first doing this.


Windows uses buffers when writing to disk. If you have sufficiently large amount of memory, you may notice what you described, i.e. Windows still writes to the disk even after the file dialog disappears. However, if you try to remove the disk using the Safely Remove disk tool, it will wait until all the buffered data are flushed to the disk.


Another interest thing I noticed due to buffering is that if the file I'm copying is already in the disk buffer, I sometimes see crazy transfer rates like 200+ MB/s, when both the source and destination disks are mechanical disks!

 

[sawfish]



Quote:
Originally Posted by gazebo  /t/1517553/problems-with-copying-large-files-to-usb-or-esata-drives-with-windows-7-x64#post_24350326


I do this all the time -- copying gigabytes and terabytes of video files from internal disks and network disks to external disks in eSATA and USB3 docks. Never noticed a problem. The reason? I always use Windows' safely remove disk tool. You should never remove an external drive without first doing this.


Windows uses buffers when writing to disk. If you have sufficiently large amount of memory, you may notice what you described, i.e. Windows still writes to the disk even after the file dialog disappears. However, if you try to remove the disk using the Safely Remove disk tool, it will wait until all the buffered data are flushed to the disk.


Another interest thing I noticed due to buffering is that if the file I'm copying is already in the disk buffer, I sometimes see crazy transfer rates like 200+ MB/s, when both the source and destination disks are mechanical disks!

^^^ This. The drives I use in docks and external enclosures include 250 GB, 500 GB, 750 GB, 1 TB, and 2 TB models, all are single NTFS partitions, and all are TrueCrypted, which is another thing that could go wrong, but never has. Several of the TB I back up are on a different machine, which my main machine accesses over the Gb network and copies to a dual USB3 dock. I back up about six TB of data onto multiple HD backup sets (one kept off-site) using SyncBackSE, and I do byte-by-byte comparisons of all the data every couple of months. I've never found a discrepancy, and I've been doing this for years. My largest files are probably 15 GB each, with a ton between 1 and 10 GB. I'm currently using USB3, but I was previously using an eSATA dock, and before that, a USB2 dock. Windows 7 x64 has been perfectly reliable for me. Before powering the drives down and removing them, I always dismount the drives in TrueCrypt, and then I use the "Safely remove" icon that appears in the notifcation area for removable drives.


While I'm using SyncBackSE, I find it hard to believe that Windows Explorer is corrupting files.

 

[ElJimador]

Interesting thread. I recently copied some large video files on to a USB hard drive for a friend and had him report back to me that some of them cut off near the end. I wonder if this might explain what happened.


BTW I always use the safe removal tool also but if the message comes back that the device can't be removed because a program or process is still using it, and I can't see any evidence that's the case, I always shut down and then remove it. So if a file was still copying at that time would it just be clipped? I would think I'd get a warning in shutdown that something was still running instead but I don't recall in that situation ever seeing that.

 

[DotJun]

It's because the copy command uses buffering along with windows doing all sorts of other things like RDC for instance. Try using an unbuffered copy like xcopy /j and you'll see the behavior change.

 

[gazebo]

I just checked some 30+ files I backed up to a 1TB drive on eSATA dock. All the files are over 5GB, and all are perfect copies of the original (according to their MD5 hashes). My OS is also 64-bit Windows 7.


With regards to eSATA, I wonder if the folks who experienced file corruption (assuming they in fact performed the Safely Remove disk routine before removing the disk) had their so-called "eSATA" implemented with something like this bracket . From a Microsoft technical paper I read a while back (I will post if I can find it), I got the impression that a real eSATA port is not the same as extending the internal SATA port to a back plate; there are also other factors (IIRC, electrical) that could confuse the OS. IIRC the paper mentioned some guidelines for OEMs. I was also under the impression that some mobo manufacturers did not properly implement eSATA, and Windows mistaken them for internal drives, and thus the logics for flushing the buffers before an external disk removal are skipped.

 

[gazebo]

Found the paper: http://msdn.microsoft.com/en-us/windows/hardware/gg463521.aspx

 

[Vlad Theimpaler]



Quote:
Originally Posted by gazebo  /t/1517553/problems-with-copying-large-files-to-usb-or-esata-drives-with-windows-7-x64#post_24352561


Found the paper: http://msdn.microsoft.com/en-us/windows/hardware/gg463521.aspx

Nice contribution to the topic, gazebo. Great information, thanks a bunch!


*EDIT*

Well, just read that paper, and it seems that the HTPC/DVR I mentioned in my first post is not configured correctly for eSATA w.r.t. "safe removal" of external storage devices in Windows 7. The eSATA port on the HTPC/DVR is on a front panel accessory for the PC case (mounted in 3.5" bay). The eSATA jack connects to the motherboard via a 20-wire harness that plugs into the mobo's 20-pin front panel eSATA header block. That eSATA port, along with all the native Intel SATA ports (Z68 chipset), are controlled under Intel's Rapid Storage Technology (RST) driver and control software for Windows 7 x64.


With this configuration, the HTPC/DVR has never had the "safe removal" icon in the Windows 7 system tray for the eSATA port. I just assumed that Intel RST was dismounting the eSATA drive properly if I turned off that device's power.


But after reading the paper, I now know that Windows 7 is designed to treat eSATA like USB w.r.t. having a "safe removal" icon in the desktop system tray. Since my HTPC/DVR doesn't have it, it is not configured properly. Since it is not configured properly, I should probably consider myself lucky that my eSATA data transfers have not turned out worse than I have experienced so far

Thanks again, gazebo, for my revelation-of-the-day

 

[gazebo]

Try uninstalling the Intel chipset/disk driver, and use the default Microsoft AHCI driver. If your motherboard's BIOS does not go out of its way to disable the hot-swappable feature of internal SATA ports that are configured for AHCI mode, then your "eSATA" drive should show up as removable.


If the above works, you will also see your other internal drives (other than the OS drive) as removable. To prevent accidental removal of these drives, you may want to set the registry setting recommended in the Microsoft paper.

 

[fatherom]

I am DEFINITELY experiencing this problem. I will post a thorough write up of what I've encountered and what I've learned. Right now, I have to head home...more tonight.


Chris

 

[trans_lux]

Ops didn't see you posted this already.


Don't wast your time with MS file manager for moving lots of or large files.

Use TeraCopy. http://codesector.com/teracopy

You'll be amazed at how much faster things move.

Let me know your results.

 

[fatherom]

Funny you should mention TeraCopy. The TechNet thread linked to by the OP has many people saying that using TeraCopy is an effective workaround. I plan to install it tomorrow.


Did want to chronicle my last two weeks of research, though. It may benefit others...


I have a Dell XPS 8300 running Windows 7 64-bit. I have about fifty TB of hard drive space, yet I currently manage all that space "manually", using eSATA docks and connecting drives when I want to copy stuff to them. I keep double (and sometimes triple) redundant backups of everything. Some of the drives are for home theater media, others are for personal stuff (photos, docs, home movies). For home theater media, I use a small SSD for playback from my Oppo/Dune, so most of my drives sit on the shelf just holding data, rarely actively used themselves.


Up until recently, I had a fairly aggressive hard drive retirement strategy. If a drive was older than 3 years (regardless of use), I bought a new drive, transferred the contents to it, and "retired" the old drive, which involves sticking it in a box somewhere with all the old data still on it. I recently realized this was too aggressive (and costly!), so I decided to catalog all my drive's actual use time (using the 'Power On Hours" SMART attribute) and use a drive's use time as the criteria for retirement.


So, a couple weeks back I started the process of going through all my drives, checking SMART status, and running MD5 checksums over all the contents of all the drives (I had done some minimal MD5 checks in years past on a subset of my files, but nothing as extensive as all my drives). I plan to do this about once a year going forward. This allows me to periodically spin up the drive, make sure there are no mechanical issues, and generate a fresh MD5 checksum for the drive's contents. This is so I could make sure my data was maintaining its integrity over the years as my drives sit on the shelf, mostly unused.


So, I did a diff between my main home theater media drive and its backup. This is a set of 2 4TB drives, each containing the same content (MKVs, Blu-ray folder structures, and Blu-ray ISOs mainly). I had more than a dozen files that were different. After using a hex editor and doing binary differences between the files, I realized I had two forms of corruption going on.


The first form of corruption was due to my own impatience. I noticed several files that were fine on one drive, but that same file on the other drive had tons of zeros at the end. The file sizes, etc, were all the same...but the file on one of the drives was "truncated" internally and filled with zeros from some point in the middle of the file all the way to the end of the file. Turns out I had pulled the drive from the dock prematurely; Windows claimed the file copy was done but the write cache was still going, apparently. This is similar to what the OP had experienced. I was able to intentionally reproduce this problem the other night, and now I know going forward to set Windows to "optimize for removal" for all my external drives and MAKE SURE to use the "safely remove hardware" icon before pulling the drive.


Now the second form of corruption is EXACTLY what people are describing in the TechNet thread cited by the OP. I noticed small differences between files that occurred randomly in the middle of the file. These differences were always a chunk of bytes, where the number of bytes in the chunk is divisible by 4096 (the default allocation unit size I use when formatting my NTFS volumes). Sometimes, one version of the file would have all zeroes in the chunk that was different. Sometimes, there would be what appears to be legit data in that chunk. Watching both versions of the file in VLC (or on the Oppo or Dune) would not show anything visibly or audibly noticeable...remember, we're talking about something like 60kbytes (for example) in the midst of a 8GB MKV file. That's what makes this problem so insidious. It's not something that manifests itself very readily. And how many people are doing CRC checks on all their files over the history of their drives and have doubly redundant backups to go back and do historical research on? My guess is this problem is more widespread than people realize, because they haven't been burned by it in terms of what this corruption means for the usability of the corrupt files.


Basically, the people experiencing this problem in the TechNet thread seem to be chalking this up to the NTFS driver in Windows 7 having a problem. There's all sorts of speculation about USB interactions, buffer problems, Dell computers specifically, etc. I would suggest reading that whole thread if you've noticed problems copying large files to external NTFS volumes. Windows 7 does seem to be the main commonality between all the people having a problem. TeraCopy does seem to be a decent workaround...it integrates into the Windows Explorer shell and can handle all copies, basically disabling the native Windows copy mechanism. It uses a different form of buffering that doesn't seem to be vulnerable to this issue. And it has built in CRC checking if you desire.


One lesson from all this is it never hurts to keep redundant backups of things. Because I have, I at least have a functional copy of every file, regardless of whether it happens to be corrupt on one of my drives.


Hope all this info may help some others out there,


Chris

 

[techmattr]

I haven't read everything in this thread but the issue should only be due to write caching. With external devices you have two options. Quick Removal (default) or Better Performance (write caching). If you have write caching enabled then you run the risk of data loss. If you have Quick Removal selected AND you use the safe remove option you don't run the risk of data loss.

 

[fatherom]



Quote:
Originally Posted by techmattr  /t/1517553/problems-with-copying-large-files-to-usb-or-esata-drives-with-windows-7-x64#post_24480196


I haven't read everything in this thread but the issue should only be due to write caching. With external devices you have two options. Quick Removal (default) or Better Performance (write caching). If you have write caching enabled then you run the risk of data loss. If you have Quick Removal selected AND you use the safe remove option you don't run the risk of data loss.

That's true for the the first form of corruption I've experienced. But the second form is a problem with the NTFS driver in win 7. If you have some time, skim the technet thread the OP mentioned. It's a real subversive problem and not everyone experiences it. I sure have. I've even had this corruption on USB drives that were set to quick removal and after clicking on safely remove hardware.

 

[Aleron Ives]

This doesn't happen when copying files between internal SATA drives, only when using USB or eSATA HDDs?


*edit*


According to the massive technet thread, quick removal mode actually makes the problem worse, not better. A few users said that switching to the "Optimize for Performance" mode made the errors go away. There was some speculation that the bug lies in W7's write-through cache for quick removal, and using the full cache from performance mode bypasses the bug, as does bypassing the cache completely with programs like TeraCopy.

 

[fatherom]



Quote:
Originally Posted by Aleron Ives  /t/1517553/problems-with-copying-large-files-to-usb-or-esata-drives-with-windows-7-x64#post_24480644


This doesn't happen when copying files between internal SATA drives, only when using USB or eSATA HDDs?


*edit*


According to the massive technet thread, quick removal mode actually makes the problem worse, not better. A few users said that switching to the "Optimize for Performance" mode made the errors go away. There was some speculation that the bug lies in W7's write-through cache for quick removal, and using the full cache from performance mode bypasses the bug, as does bypassing the cache completely with programs like TeraCopy.

I have never experienced any problems with internal SATA drives, nor (seemingly) have the users on the technet thread.


Interesting about quick removal making it worse. I read a lot of the technet thread, but skimmed parts, so I missed that. I have had the corruption happen in both instances...quick removal and optimize for performance. It seems the only true "fix" is to use something like TeraCopy.

 

[techmattr]

This could also have to do with whatever storage controller is being used in the external enclosure or USB drive. The only time I've ever had corruption occur on an external drive was when I was using a cheap USB stick. I also have one that went through the wash that seems to get corrupted quite often. I'd be curious to know if someone seeing this issue with Windows 7 can reproduce the same issue with the same hardware but with Windows 8.

 

[fatherom]



Quote:
Originally Posted by techmattr  /t/1517553/problems-with-copying-large-files-to-usb-or-esata-drives-with-windows-7-x64#post_24481284


This could also have to do with whatever storage controller is being used in the external enclosure or USB drive. The only time I've ever had corruption occur on an external drive was when I was using a cheap USB stick. I also have one that went through the wash that seems to get corrupted quite often. I'd be curious to know if someone seeing this issue with Windows 7 can reproduce the same issue with the same hardware but with Windows 8.

This problem is not affecting everyone with Windows 7, to be sure. But it's affecting enough people that it's been noticed. I have had this corruption happen with Western Digital self-powered passport drives, Seagate external drives, and bare drives (WD and Seagate) in both a USB dock and an eSATA dock. So, I doubt this is something to do with the hardware of the external device since I've reproduced this corruption across many different forms of hardware.


Many in the technet thread said they've never seen corruption, so it's definitely subtle mix of factors...potentially hardware, software, drivers, etc. I wouldn't be surprised if only Dell PCs have this issue (although, that's admittedly a hunch).


I did some searching last night and have not found anyone mention this problem under Windows 8.

 

[fatherom]

Another piece of anecdotal info:


I keep meticulous records. I keep a journal where I write down when I transfer files and to which drive I've transferred them to.


In the technet thread, the original poster found that using the computer for other tasks seem to make the problem happen more often. He even speculated that just the act of using the USB keyboard or mouse could cause it to happen. He also noticed that he NEVER got corruption when connecting to his PC via Remote Desktop (seems to imply that local hardware use has something to do with it).


For my files that are corrupt, I was physically sitting at my computer doing other things while the copy was taking place. Sometimes surfing the net, other times copying files to a different external USB device. Sitting in front of my computer is actually a rare thing for me...I tend to be remotely connected from a laptop downstairs (using a program called Radmin). That may be why I have such little corruption across my fifty TB. Without realizing it, I was mitigating the problem by not sitting in front of my computer but doing most of my stuff over a remote connection. Who knows...again, this problem is subtle and hard to pin down, to say the least.


I realize this all sounds strange, and many people may never experience a problem. But I would wager that if more people were doing CRC checks on all their files over the course of years like I am, then it's possible more people have run into this and never been aware of it.


Chris