If you had to start over, how would you handle storage? - AVS Forum
Forum Jump: 
 
Thread Tools
post #1 of 8 Old 04-04-2013, 02:17 PM - Thread Starter
AVS Special Member
 
Chronoptimist's Avatar
 
Join Date: Sep 2009
Posts: 2,559
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 203
I have been gradually upgrading my PC over the years, which has now become a do-it-all box - Gaming, Blu-ray & DVD, Music playback, Browsing etc. and it is now at the point where I have reached the limit of how many hard drives my case will take without adding more hardware to it. (SATA card, removable drive bays etc.)
It was not built as a HTPC, but rather a (moderately) quiet gaming PC, that has ended up taking over all of my home entertainment - especially since purchasing JRiver Media Center recently, which I am very happy with. It's now the only device that is even hooked up to my TV.

I have a couple of 4TB disks in my machine now, a few 2TB drives, and a few terrabytes in old disks sitting around that are perfectly functional, but aren't actually hooked up.
It seems that hard drive capacity has remained unchanged for a couple of years now, and I haven't seen announcements for anything over 4TB in size, so the only solution is more disks rather than upgrading disks - replacing the 2TB drives with 4TB ones is not cost effective.
Previously I replaced 500GB disks with the ones that are in there now - if we had 8+TB disks available today, I might have considered replacing the 2TB drives with them.


Because this was not originally built for HTPC duties in mind, I have been a bit carefree with it. I'm running a regular Windows install, and the drives are just hooked up normally, with files spread out over them in no particular order.
I'm not running RAID or any other kind of backup system, because I was not expecting it to completely replace all my other devices, and I figured that I still have all the discs in storage and could re-rip them again if a drive ever did fail.

But now that I am using these high capacity disks, it's getting to the point that a single drive failure would be a significant time investment, rather than when I only had a couple of 500GB drives in there. (which only store a handful of discs if it's films rather than music)

With Haswell on the horizon, and presumably new GPUs from Nvidia on the way (the mobile 700 series was just released) I'm going to be building a new gaming PC in the upcoming months, which seems like a good opportunity to get storage sorted out at the same time.


So for those of you that have very large libraries now (over 10TB) how are you handling data storage and redundancy? What would you do differently if you had the opportunity to start over today?

My biggest concerns are data access speeds, and how reliable things like RAID actually are. I've had a QNAP NAS device in the past, and found it to be rather slow, which has put me off the idea of networked storage for a long time. It seems that on the consumer side, we're still stuck with gigabit ethernet, which seems like it would be a lot slower than internal drives.

And I've heard horror stories about one drive dying and being replaced, the array taking days to rebuild, only to have it fail and losing the data on all drives.
If I'm moving to some kind of networked storage, having days of downtime if a disk dies is not acceptable.
Chronoptimist is offline  
Sponsored Links
Advertisement
 
post #2 of 8 Old 04-04-2013, 02:25 PM
AVS Special Member
 
TornadoTJ's Avatar
 
Join Date: Jan 2008
Location: Dallas, TX
Posts: 3,915
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 33
If were starting over, I'd use very fast drives. The gigabit Ethernet being slower than the drive isn't the issue. My server does a lot more than just serve video files. There are lots of other things asking for disc time, and it can all get bottlenecked. I have moved some functions off to separate drives, but it still can be an issue. My green Seagate drives just aren't fast enough for what all I'm throwing at the server. By now you can tell I'm talking about on the server side, I don't like keeping all my storage on any of the machines I touch. The server is there 24/7 to handle whatever requests come from the various computers on my network.
TornadoTJ is offline  
post #3 of 8 Old 04-04-2013, 02:50 PM - Thread Starter
AVS Special Member
 
Chronoptimist's Avatar
 
Join Date: Sep 2009
Posts: 2,559
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 203
Quote:
Originally Posted by TornadoTJ View Post

If were starting over, I'd use very fast drives. The gigabit Ethernet being slower than the drive isn't the issue. My server does a lot more than just serve video files. There are lots of other things asking for disc time, and it can all get bottlenecked. I have moved some functions off to separate drives, but it still can be an issue. My green Seagate drives just aren't fast enough for what all I'm throwing at the server. By now you can tell I'm talking about on the server side, I don't like keeping all my storage on any of the machines I touch. The server is there 24/7 to handle whatever requests come from the various computers on my network.
Even just locally, I have been finding this to be the case as well - but not just with my green disks. To be honest, I haven't noticed much difference between my 4TB WD Green drives and the 7200 RPM 2TB ones. But perhaps that's due to the increased density.

My system does not support Intel SSD caching, and that seemed like it would be the solution for this when I upgraded - at least for local storage - as I had planned on buying a 256GB SSD and using it as a 64GB cache (maximum allowed) for four disks, but it seems that they only allow you to cache one HDD per SSD.
Even though I had initially discounted them, it seems that hybrid drives may actually be the way forward - but there don't seem to be any high capacity 3.5" hybrid disks yet, the manufacturers seem to be focusing on notebook drives right now.


But I am curious about your comment about gigabit ethernet not being the bottleneck in your system. The theoretical maximum transfer rate for it is 125MB/s, and in reality you're probably going to see about 10-20% less than that. A single hard drive in my system can have transfer speeds exceeding that rate, and I would expect performance to be higher when working in RAID.
That might be fine for streaming video, but seems inadequate if you're using it for data access. Is 10GbE the only other option? It seems that 10GbE hardware is all pro-grade gear right now, and far too expensive.
Chronoptimist is offline  
post #4 of 8 Old 04-04-2013, 04:00 PM
AVS Special Member
 
TornadoTJ's Avatar
 
Join Date: Jan 2008
Location: Dallas, TX
Posts: 3,915
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 33
There's other overhead going on. I don't know why or what it is (I haven't looked into it) but sometimes my disks will just start thrashing like crazy. I'm not talking the boot drive, it's the data drives. I don't know what's going on, but it kills throughput. Also, I have other processes that go on locally that take up disk time, and those impact the disk speed (availability) to devices that are asking for files for streaming, transcoding, whatever. So my comment has to do with those local processes killing my drives.
TornadoTJ is offline  
post #5 of 8 Old 04-05-2013, 02:34 PM
AVS Special Member
 
user4avsforum's Avatar
 
Join Date: Mar 2006
Location: California
Posts: 1,180
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 11
Quote:
Originally Posted by Chronoptimist View Post

So for those of you that have very large libraries now (over 10TB) how are you handling data storage and redundancy? What would you do differently if you had the opportunity to start over today?
I have 15TB of storage between 2 HTPCs and a WHS. Although I am a bit shy of a 10TB library, even if I was over 10TB I would use the same basic storage method I am now.

- I do not use any mirroring or RAID on any system. I am not against it, just don't have any need to.
- My wired network is gigabit, only notebook PCs are wifi.
- Boot drives on all PC systems are backed up to the WHS. The WHS boot drive is backed up to a secondary drive on the WHS.
- I use SyncToy to maintain duplicate copies of media files I do not want to lose.

- Recorded TV is stored on the HTPC where it is recorded and shared across the network.
- Pictures, Music, and Home Videos (post production) are sync'ed daily across HTPCs, Server, and home PCs so all systems have a copy.
- The ripped disk/movie library is stored on the server and sync'ed daily to a secondary drive on the server to create a backup.

Why this setup?
- I consider Recorded TV to be disposable, if I lose it no big deal. If there is a recording I want kept permanently I move it to the WHS movie library.
- RAID is great for drive failures however it does not gaurd against accidental deletion or many other types of file corruption. I would never trust my library to a system that only provides protection against drive failures.
- By keeping local copies of all of the digital photos and videos they are always accessible on every system, even on notbook PCs while traveling. This also acts as a great backup system while also allowing me to add new content from any PC and it shows up everywhere.
- If I have some type of catastrophic failure I am confident that I can easliy pull out any of my drives and plug them into almost any other system and recover my data.
user4avsforum is offline  
post #6 of 8 Old 04-05-2013, 02:51 PM
AVS Special Member
 
TornadoTJ's Avatar
 
Join Date: Jan 2008
Location: Dallas, TX
Posts: 3,915
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 33
I forgot to mention, I'm also not running any RAID. I'm duplicating what is on my PCs to the WHS using the built-in function of the WHS.
TornadoTJ is offline  
post #7 of 8 Old 04-05-2013, 07:53 PM
Advanced Member
 
Davecraze's Avatar
 
Join Date: Nov 2004
Location: Irving, Texas
Posts: 758
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 4 Post(s)
Liked: 17
38TB (24TB used) in eight bay enclosure units attached to my media server. Nothing fancy - no RAID striping or mirroring.
I use flexraid to do parity protection on all the drives and flexraid to drive pool 20TB of that for movies. Serves four media streamers and various iOS devices.

All my 19 drives are 2TB as my server still runs win xp 32. I guess i could do a clean install to jump to 64bit but it is easier for me to just keep adding
additional enclosures if required.

I would not change anything else if I started over - ok, maybe use the lan li pc343b case to have all the drives in one case.
Davecraze is offline  
post #8 of 8 Old 04-05-2013, 08:01 PM
Member
 
kingpin748's Avatar
 
Join Date: Jan 2009
Posts: 81
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 14
I just wish Server 2012 Essentials had come along sooner. I switched about 6 weeks ago while adding 2 drives. I'm at 20TB and am enjoying how everything just seems to work. WHS V1 was a hassle to do anything usually.
kingpin748 is offline  
Reply Home Theater Computers

User Tag List

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off