I have a WHS server for my media serving needs with just under 10TB of raw storage w/duplication turned on, I have 4.7 TB of data. It has 7 storage drives and 2 drives mirrored for OS. It seems every time I put another drive in there, it becomes full in no time. Does anyone else have this problem? Am I a media hoarder, like that hoarding show on TLC?
Media Server - Why am I always running out of Space?
-
Now Available: Tech Talk Podcast with Scott Wilkinson, Episode 19 Click here for details.
1 - 20 of 90 Posts
I found the problem
![]()
![]()
You may want to start compressing your shows and editing out commercials. If these are movies, it might be time to pick which ones you really need to have on the server. I had the same problem, and then I realized I didn't need the blueray of band of brother ripped to my server. It was over 200GB, and I relaized it would be years before I wanted to watch it again.
I found the problem
You may want to start compressing your shows and editing out commercials. If these are movies, it might be time to pick which ones you really need to have on the server. I had the same problem, and then I realized I didn't need the blueray of band of brother ripped to my server. It was over 200GB, and I relaized it would be years before I wanted to watch it again.
Joined
·
2,043 Posts
No, but it is the perfect replacement for WHS duplication as a method of recovery from HDD failures.
While replacement of media is not impossible, it is extremely inconvenient and time-consuming, and warrants a bit of extra expense/hardware and 1 - 2 parity drives.
No, but it is the perfect replacement for WHS duplication as a method of recovery from HDD failures.
While replacement of media is not impossible, it is extremely inconvenient and time-consuming, and warrants a bit of extra expense/hardware and 1 - 2 parity drives.
Joined
·
913 Posts
i agree with the poster who stated that RAID is unnecessary in HTPC applications. I would further what he said, by noting that RAID increases energy consumption, increases hard drive usage, adds the extra complexity of a hardware or software RAID controller, and in fact INCREASES your risk of catastrophic data loss.
In the AVS forums, you can find numerous reports of people losing entire RAID arrays (multiple hard drives of data, all at once) either through operator error, or failure of the RAID controller card.
The average corporate RAID user utilizes HVAC, cooling, maintanence, and monitoring services to maintain data integrity, that are impractical in the home setting.
My guess is that most htpc RAID users don't even monitor hard drive temps, much less use redundant power supplies, UPS, and redundant cooling systems, and alarms.
It is unheard of to lose multiple hard drives simultaneously when used in a JBOD setup.
Hey, you want to use RAID, go for it! But don't for a minute believe it's necessary, or even desirable if you just want to store movies.
In the AVS forums, you can find numerous reports of people losing entire RAID arrays (multiple hard drives of data, all at once) either through operator error, or failure of the RAID controller card.
The average corporate RAID user utilizes HVAC, cooling, maintanence, and monitoring services to maintain data integrity, that are impractical in the home setting.
My guess is that most htpc RAID users don't even monitor hard drive temps, much less use redundant power supplies, UPS, and redundant cooling systems, and alarms.
It is unheard of to lose multiple hard drives simultaneously when used in a JBOD setup.
Hey, you want to use RAID, go for it! But don't for a minute believe it's necessary, or even desirable if you just want to store movies.
In my opinion, the best option for media server data protection is FlexRAID. It is free and does not "place your data at risk" as some can argue as a striped raid does. Your data would be stored as it is now, on individual disks. But rather than duplicating all that data, only 1 HD equal to the largest HD in your 'parity' array is required (if you have 3 1TB drives and 2 2TB drives, one of the 2TB drives has to be your parity). This redundancy will protect you from up to 1 single hard drive loss, but will only cost you the drive space of 1 HD. And if you lose more than 1 HD, then you only lose the data on those failed HDs and not all data across your entire array. Another advantage is since the HD is not part of a striped array, it can be placed in any computer and be read.
Plus you don't have to buy any additional hardware or change your OS. You could install it tonight and be protected tomorrow morning. It's biggest downfall is the learning curve of how to impliment it, but I'm running it and familiar with it and could always offer some help.
Check out www.openegg.org for more info.
Plus you don't have to buy any additional hardware or change your OS. You could install it tonight and be protected tomorrow morning. It's biggest downfall is the learning curve of how to impliment it, but I'm running it and familiar with it and could always offer some help.
Check out www.openegg.org for more info.
Joined
·
259 Posts
I disagree. I really don't want to rip 250 bluray movies at an hour a piece back to the server if I lose a drive. my time is worth more than the extra drives cost.
Just my opinion.
I disagree. I really don't want to rip 250 bluray movies at an hour a piece back to the server if I lose a drive. my time is worth more than the extra drives cost.
Just my opinion.
I'm of the same opinion. While I still have all of my DVD's and Music CD's in physical media, I have the folders duplicated cause I'm not going thru that project again. Same with my pictures, of course.
But when I am able to get the Ceton card or the HD Homerun, any TV shows I record, I won't consider critical and won't duplicate those folders.
I'm of the same opinion. While I still have all of my DVD's and Music CD's in physical media, I have the folders duplicated cause I'm not going thru that project again. Same with my pictures, of course.
But when I am able to get the Ceton card or the HD Homerun, any TV shows I record, I won't consider critical and won't duplicate those folders.
Wow you must be doing some heavy compressing?...I'm only able to fit about 80-90 full-res MKV HD-ripped Movies on a 2TB drive. All my drives are simple NTFS volumes that are then pooled with MediaBrower with their UNC share name...I can move the drives from box to box and they work fine. If I lose one then I have to re-rip about 80 movies it would take about 1 minute of setup per movie with my GUI then walk away for an hour or less....If it happened I wouldn't be happy but it’s not the end of the world, it’s for sure not worth losing much sleep over or doubling the out of pocket cost and energy with more spindles.
Mike
Wow you must be doing some heavy compressing?...I'm only able to fit about 80-90 full-res MKV HD-ripped Movies on a 2TB drive. All my drives are simple NTFS volumes that are then pooled with MediaBrower with their UNC share name...I can move the drives from box to box and they work fine. If I lose one then I have to re-rip about 80 movies it would take about 1 minute of setup per movie with my GUI then walk away for an hour or less....If it happened I wouldn't be happy but it’s not the end of the world, it’s for sure not worth losing much sleep over or doubling the out of pocket cost and energy with more spindles.
Mike
Joined
·
1,165 Posts
I to agree than RAID5 or RAID6 is really unneccessary for home use. A striped array is best done with good hardware controllers and enterprise rated drives. The main reason is for speed which is almost never required for media storage. Some people might like having the speed just because, but it's not a requirement.
I do give up the space of a parity drive in unRAID to get protection against a single hard drive failure. But, unRAID is more a group of individual drives protected by a parity drive. It does not use striping.
Flexraid can do it too. You have to be careful how you use it though. The loss of data that can happen is not simple to understand when another drive fails besides the one where you made changes. In this case, if you have a failure between a parity generation and changing data then you have hosed the equivalent amount of data on the other drive and what got hosed is not known until it's lost. For example, make changes to 10 gig of data on one drive and then another drive fails. Up to 10gig of data on this failed drive won't be recoverable. The data could damage one big >10gig file or it could damage 100 x 100meg files. It just depends on how lucky (or unlucky) you are. So, pay attention to the recommendation of using it for static data and keep the parity updated.
With Flexraid, I'm also saying drive just for simplicity. Flexraid can go deeper than the drive level when selecting what to protect.
Peter
I do give up the space of a parity drive in unRAID to get protection against a single hard drive failure. But, unRAID is more a group of individual drives protected by a parity drive. It does not use striping.
Flexraid can do it too. You have to be careful how you use it though. The loss of data that can happen is not simple to understand when another drive fails besides the one where you made changes. In this case, if you have a failure between a parity generation and changing data then you have hosed the equivalent amount of data on the other drive and what got hosed is not known until it's lost. For example, make changes to 10 gig of data on one drive and then another drive fails. Up to 10gig of data on this failed drive won't be recoverable. The data could damage one big >10gig file or it could damage 100 x 100meg files. It just depends on how lucky (or unlucky) you are. So, pay attention to the recommendation of using it for static data and keep the parity updated.
With Flexraid, I'm also saying drive just for simplicity. Flexraid can go deeper than the drive level when selecting what to protect.
Peter
It'll be nice when the developer of FlexRAID gets FlexRAID live up and running, which will basically work just like unRAID then and will maintain your parity live as data is written.
He's currently focusing on FlexRAID View. It's basically a data solution to pool all your hard drives (or folders within HD or even network shares) into a single share and have it control which drive to place the data on. It doesn't balance data like WHS, though. You set a cap on each HD and when reached it moves to the next.
He's currently focusing on FlexRAID View. It's basically a data solution to pool all your hard drives (or folders within HD or even network shares) into a single share and have it control which drive to place the data on. It doesn't balance data like WHS, though. You set a cap on each HD and when reached it moves to the next.
Joined
·
4,217 Posts
RAID isn't a backup.
RAID isn't a backup.
I don't understand your comments about increasing HDD usage or complexity. If you need 8 drives worth of space, IMO a RAID solution is simpler because you're going to need some sort of controller anyway, and RAID will get you one volume instead of 8.
And I disagree with this in practical terms, for the very reason you state just below "It is unheard of to lose multiple hard drives simultaneously". RAID or not doesn't change your likelihood of losing multiple drives.
You might, I haven't, but even if so, it's usually due to operator error (as you say, and nothing protects you from that) or using subpar equipment. There's a reason those of us who "recommend" RAID recommend doing it only in the confines of using appropriate equipment.
You don't need that in a home setup.
Becasue you don't need to.
Same is true of RAID.
unRAID would be great if you could do anything else with the box. But you can't for all intents and purposes. I have no personal affinity for RAID, other than if I'm using 1.5TB drives, I don't want to have to spend a month re-ripping movies if one drive fails. Unfortunately RAID is really the only way to get that functionality in a reliable manner in a system that does more than share files.
FlexRAID will be great if they ever get it to the bulletproof point and a config that's setup as a driver/service and not an app, so that it's operation is seamless.
I don't understand your comments about increasing HDD usage or complexity. If you need 8 drives worth of space, IMO a RAID solution is simpler because you're going to need some sort of controller anyway, and RAID will get you one volume instead of 8.
And I disagree with this in practical terms, for the very reason you state just below "It is unheard of to lose multiple hard drives simultaneously". RAID or not doesn't change your likelihood of losing multiple drives.
You might, I haven't, but even if so, it's usually due to operator error (as you say, and nothing protects you from that) or using subpar equipment. There's a reason those of us who "recommend" RAID recommend doing it only in the confines of using appropriate equipment.
You don't need that in a home setup.
Becasue you don't need to.
Same is true of RAID.
unRAID would be great if you could do anything else with the box. But you can't for all intents and purposes. I have no personal affinity for RAID, other than if I'm using 1.5TB drives, I don't want to have to spend a month re-ripping movies if one drive fails. Unfortunately RAID is really the only way to get that functionality in a reliable manner in a system that does more than share files.
FlexRAID will be great if they ever get it to the bulletproof point and a config that's setup as a driver/service and not an app, so that it's operation is seamless.
ok to see all the raid bashing. I built a 21tb raid 6 that has a UPS. What economical/practical alternative is there? You cant economically backup 21tb of data. It would cost thousands. It would probably cost more than building a redundant array which is probably what I would do next. I dont see flexraid or unraid as being any better than hardware raid as none of those are a backup either. The only actual backup that I can even see working would be tape backup and that costs thousands as well. What else is there?
Drive usage: With a striped array, all drives must spin to read or write a file. With pseudo-RAID, only the drive where the file resides must spin.
Complexity: Motherboards with 8-10 SATA ports are common. You can easily build a 10 drive server with no additional hardware needed. If you need more, 4-port cards are $20-$30. Compare that to hardware RAID cards which are generally in the $500-$800 range. Both WHS and unRAID allow you to treat all connected drives as one pool, or one volume, if you prefer.
No, but it does change how much data is lost if you do lose multiple drives. If your hardware RAID array goes down and cannot be rebuilt for any reason, you lose all data on all drives. Whether you lose multiple drives simultaneously, or faulty hardware corrupts the array, or the controller dies and the replacement just plain won't rebuild for whatever reason, you lose everything. With Hardware RAID, either you have everything, or you have nothing.
With Pseudo-RAID, this is not the case. Nothing is striped, and data is kept whole at the file level. That means that any individual drive can be pulled out of an array, connected to another computer, and the files contained on it can be retrieved. No matter what happens to a Pseudo-RAID array, any drives that are still functional can have their contents recovered. If you have 10 dirves on your server, and suddenly three die, you still have seven drives worth of data. With hardware RAID in that situation, you would have no recoverable data at all.
As others have mentioned, there are pseudo-RAID options like FlexRAID and unRAID. They give you the primary benefits of hardware RAID (drive pooling, parity-based redundancy) without the drawbacks (huge hardware cost, limited upgradability, energy inefficient, potential for catastrophic total data loss). The only thing hardware RAID has over pseudo-RAID is speed. In most cases, unless you need to serve several HD streams simultaneously, this isn't an issue for most home users.
Drive usage: With a striped array, all drives must spin to read or write a file. With pseudo-RAID, only the drive where the file resides must spin.
Complexity: Motherboards with 8-10 SATA ports are common. You can easily build a 10 drive server with no additional hardware needed. If you need more, 4-port cards are $20-$30. Compare that to hardware RAID cards which are generally in the $500-$800 range. Both WHS and unRAID allow you to treat all connected drives as one pool, or one volume, if you prefer.
No, but it does change how much data is lost if you do lose multiple drives. If your hardware RAID array goes down and cannot be rebuilt for any reason, you lose all data on all drives. Whether you lose multiple drives simultaneously, or faulty hardware corrupts the array, or the controller dies and the replacement just plain won't rebuild for whatever reason, you lose everything. With Hardware RAID, either you have everything, or you have nothing.
With Pseudo-RAID, this is not the case. Nothing is striped, and data is kept whole at the file level. That means that any individual drive can be pulled out of an array, connected to another computer, and the files contained on it can be retrieved. No matter what happens to a Pseudo-RAID array, any drives that are still functional can have their contents recovered. If you have 10 dirves on your server, and suddenly three die, you still have seven drives worth of data. With hardware RAID in that situation, you would have no recoverable data at all.
As others have mentioned, there are pseudo-RAID options like FlexRAID and unRAID. They give you the primary benefits of hardware RAID (drive pooling, parity-based redundancy) without the drawbacks (huge hardware cost, limited upgradability, energy inefficient, potential for catastrophic total data loss). The only thing hardware RAID has over pseudo-RAID is speed. In most cases, unless you need to serve several HD streams simultaneously, this isn't an issue for most home users.
as stranger mentioned above those unraid solutions are no better at potential data loss than hardware raid. It might be more expensive but it offers bonuses as well such as speed which is highly useful in many ways.
I used to burn discs. Thousands of them. But that isnt practical either with the exponential data size increases today. I have about 15tb of data to backup right now. I looked at tape backup and its EXPENSIVE as well. Maybe someone can show me a solution?
My other thought was to build an even bigger array next and duplicate the data like that. But then you have the energy usage and i have a feeling my wife has been noticing the large electric bills.... its hard to shrug that off and say it must be the lights being left on...
I used to burn discs. Thousands of them. But that isnt practical either with the exponential data size increases today. I have about 15tb of data to backup right now. I looked at tape backup and its EXPENSIVE as well. Maybe someone can show me a solution?
My other thought was to build an even bigger array next and duplicate the data like that. But then you have the energy usage and i have a feeling my wife has been noticing the large electric bills.... its hard to shrug that off and say it must be the lights being left on...
Joined
·
358 Posts
same conclusion here -- too much work or expensive to do tape or optical media like bluray. for last 2-3 years, I use 400-500 gig harddrives like giant floppies or CD-R. now even that is a hassle. so I'm now growing with more storage chassis and arrays that will hold duplicate copies of data and I will now keep my original media offsite -- in case I need a true rebuild from a total disaster like a fire.. smaller sets of data like photos, audible and music collection I still make copies to extra harddrives and have offsite copies and copies in fire safe, etc.. I'm around 10TB of data at this point..
Clearly I'm doing SAS controllers and expanders with cascading multiple drive chassis's
Clearly I'm doing SAS controllers and expanders with cascading multiple drive chassis's
Yeah thats exactly what I am planning next. I am just buying a rack and going straight for the pro stuff. I figure I can do 2 of those norco 24 bays and just put them on top of each other duplicating all the data. If I need to expand beyond that I can buy larger capacity drives.
When you are talking massive amounts of data I just dont believe there is a viable alternative.
When you are talking massive amounts of data I just dont believe there is a viable alternative.
1 - 20 of 90 Posts
