or Connect
AVS › AVS Forum › Video Components › Home Theater Computers › BUILD LOG: 48 Terabyte Media Server
New Posts  All Forums:Forum Nav:

BUILD LOG: 48 Terabyte Media Server

post #1 of 820
Thread Starter 


Welcome to my 48 Terabyte Media Server and 20 Terabyte Backup Server Build Log. This project spans three phases - the first phase created a 24 Terabyte server, the second phase expanded storage to 48 Terabytes, and the third phase (still in progress) is building a 20 Terabyte backup server at the lowest possible cost.

Purpose: In December 2007, my growing data storage requirements had reached a logistical mess: six low-end tower servers with a few terabytes of storage each to store raw HDV cam footage, hundreds of multi-track audio recordings and thousands of DVD backups. It had become difficult to have my data spread across so many computers and having to keep track of what data was located where. I needed a "consolidated" storage solution, and name brand turnkey solutions at 24 Terabytes were unrealistic with their $50,000+ pricepoint. I didn't like the idea of buying a tower case and filling up all the 5.25" bays with "5-in-3" SATA backplanes - I wanted a case pre-built for storage. Along came the Supermicro 24-Bay SC846TQ-R900B case in January 2008, so I created a shopping list of server-class components that had high value-to-cost ratios. The server not only needed to store all my data but transcode occasional large batches of HDV and MPEG2 footage to H.264, which meant the need for decent CPU and RAM.

Build Status: [Updated 7-26-2008]

Phase 1 is completed. This phase began in January 2008 and consisted of building the initial 24Tb server.

Phase 2 is completed. This phase began in March 2008 and consisted of adding an additional 24Tb of externally attached storage.

Phase 3 is in progress. This phase is a "backup server" and began in July 2008 with the arrival of the cost effective Norco 4020 case, which will add 20Tb of "backup storage" for the most important data on the 48Tb system. Other components (motherboard, multi-port SATA card, etc) being determined. This phase may be of interest to more people based on feedback I've received, since the goal of this phase will be to reach 20Tb of storage at the LOWEST TOTAL COST. The reasoning between a 20Tb backup server for a 48Tb media server is explained in the FAQ section.

NOTE: If you're thinking about a large storage solution of your own, please see the Frequently Asked Questions at the bottom of this post for recommendations on build parts, feature questions, etc.

Phase 1 - Parts List:

Motherboard: Supermicro X7DWN+ Dual 771 Intel 5400 Seaburg Extended ATX Server Motherboard
Case: Supermicro SC846TQ-R900B 4U Rackmount 24-Bay Hot Swap Chassis (dual 900w redundant High Efficiency PSU's)
CPU: 2 x Intel Xeon E5410 Harpertown 2.33GHz 12MB L2 Cache Socket 771 Quad-Core (8 cores total)
RAID Controller: Areca 1280-ML 24 Port PCI-E SATAII Raid Controller with MiniSAS to SATA (SFF-8087) 1x4 fanout cables
RAID Controller: 2GB Cache Memory Upgrade Kit for Areca controller (third party)
Harddisk: 1 x WD Raptor 150Gb HD for system volume (drive "C") no redundancy, I dump a drive image with Acronis periodically
Harddisk: 24 x Hitachi 1TB SATA Hard Drive, 32mb cache, part# 0A35155 (drive "D" in RAID6)
Memory: 16 x Kingston 2GB DDRII FB-DIMM DDRII-667 Memory Modules (total of 32GB Ram)
O/S: Microsoft Windows 2003 Server x64 Enterprise (64-bit)

Phase 2 - Parts List: (This chassis is stacked on top of the first one, and SFF-8087 cables run between them)

Case: Supermicro SC846TQ-R900B 24 Hot Swap SATAII Chassis (dual 900w redundant HE PSU's)
RAID Controller: Areca 1280-ML 24 Port PCI-E SATAII Raid Controller with MiniSAS to SATA (SFF-8087) 1x4 fanout cables
RAID Controller: 2GB Cache Memory Upgrade Kit for Areca controller (third party)
Harddisk: 24 x Seagate ST31000340AS 1TB drives, 32MB cache (drive "E" in RAID6)

Phase 3 - Parts List:

Motherboard: TBD (to be determined)
Case: Norco 4020 4U Rackmount 20-Bay Hot Swap Chassis (1000 Watt powersupply being determined)
CPU: TBD
RAID Controller: TBD
Harddisk: TBD
Memory: 2 x Kingston 2GB DDRII FB-DIMM DDRII-667 Memory Modules (total of 4GB Ram)
O/S: TBD

Gallery (Phase 1)



Gallery (Phase 2)



Gallery (Phase 3)

(Coming Soon)

Benchmarks: This HDTach benchmark screenshot was taken during the build when the array was still RAID5 and had only 16 drives, just to give an idea of the Areca 1280ML's performance. As you can see it can burst to over 1 GB/second with sustained reads of near 800MB/s. Read performance goes up almost linearly with each additional drive added to the array. In the meantime I have 24 drives in RAID6, and another benchmark will be added to show I/O for a 24 drive array in RAID6.



Formatted Capacity: This screenshot shows formatted capacity of a 22Tb partition in Windows. Again, 2 drives worth of space lost to RAID6 overhead.



CPU in Task Manager: This screenshot shows Mainconcept Reference encoding HDV footage to H.264, utilizing all 8 Xeon cores pegged to almost 100%. Since this screenshot was taken I've warmed up to x264 much more, which makes even more efficient use of multiple cores than Mainconcept Reference.



Frequently Asked Questions:

Q: What are the best parts to build a large storage server?

Case: I no longer recommend the Supermicro 24-bay case, because it's noisy (power supply fans and internalfans) and at the $1000 price point it's no longer competetive with the Norco 4020 that came out July 2008, with it's amazing $289 pricepoint for 20 x 3.5" hot swap bays. The ability to add a quiet power supply also means being able to run it much quieter than a Supermicro 24 bay case.

RAID card: In January 2008 the Areca 1280ML card was the only choice for a 24-port SAS/SATA card to match the Supermicro 24-bay case. Meantime, a new generation of cards have appeared from both Adaptec (5 series) and Areca (1680ix series), both which use a new dualcore 1.2Ghz Intel IOP chip that makes for very fast array operations that require parity calculation (i.e. array build, rebuild, write performance to Raid5 and Raid6 arrays, etc). Adaptec offers 5-series cards from 4 ports to 28 ports. Areca offers a similar lineup. These cards are expensive, but a good investment but a good investment. I'm currently exploring the area of more inexpensive ($125) 8-port cards from Supermicro and others.

Harddisks Initially I bought Hitachi 1Tb drives which were highly rated, however as of July 2008 there are better options. I generally recommend Seagate 1Tb drives due to their 5-year warranty. You can't go wrong with Seagate.

Q: What size harddisk do you recommend? What about new 1.5Tb and 2Tb drives coming out?
A: When you create a raid array, you want to standardize on only one make and model of harddisk, and then keep buying only that model so they are matched in the array. I feel 1Tb drives are a future-proof "sweet spot" right now, regardless of 1.5Tb drives arriving in Q4 2008 and 2.0Tb drives arriving in 2009. The larger 1.5Tb and 2.0Tb drives will carry a premium just as the 1Tb drives did when they first came out. Standardizing on the 1Tb size for your storage array now means they'll only continue to get cheaper when bigger drives come out. Waiting on 1.5Tb drives will result in diminishing returns on your money. 1Tb is big enough for most people as an array drive, since for example 20 x 1Tb drives (20Tb) versus 20 x 1.5Tb (30Tb) = difference of 10Tb, and MOST people don't have enough media to fill up more than 20Tb in the next 12-24 months anyway, so waiting on 1.5Tb drives wouldn't make sense for most people. Note I didn't feel the same way about using cheaper 500Gb and 750Gb drives instead of 1Tb drives when I was deciding on what to standardize on. To me, 1Tb *was* worth waiting for, as it's a "sweet spot" in terms of space, and a nice even number that makes space calculations easy (1 drive = 1 slot = 1Tb). With 1Tb drives as low as $149 now, when Seagate's first 1.5Tb drive arrives later this year it will be MSRP $399 and sell for $299-$349. At that point 1Tb drives will be $99-$129. For the next 12-24 months, you can't go wrong with standardizing on 1Tb now.

Q: How are you backing up a 48 Terabyte server with only a 20 Terabyte backup server? Why do you have to back it up at all since the 48 Terabyte server already features redundancy in the Raid6 arrays?
A
: The 48Tb system is nowhere near full capacity. Additionally, there are only about 10 Terabytes (and growing) of data I have that is irreplaceable. Some people question the need for backup of a redundant system like a Raid6 array. The simple answer is that a Raid6 array and its controller are not likely to fail, but with the cost of harddisk based storage at historical lows, it's worth it to me to have a second copy of my most valuable data, regardless of redundancy in the arrays.

Q: Does the Areca 1280ML raid controller support staggered spin-up and/or drive spindown due to inactivity?
A: Yes, it supports staggered spin-up. Areca is also one of the only manufacturers that support array spin-DOWN on their cards. Staggered spinup can be set in intervals of .4 seconds through 6 seconds. Inactivity spindown can be set from 1-60 minutes, or disabled. Screenshot of the options are in the picture below - a snippet from the web-based configuration GUI.



Q: Do the dual 900 Watt power supplies of the Supermicro 24-bay case draw a lot of power?
A: Not as much as you'd think, these power supplies are "N+1", meaning that even when both are plugged in, there is only one power supply worth of draw (except for a tiny bit of power that spins the second PSU's fan). With both power supplies plugged into a power strip, and that power strip plugged into a watt-meter, and with all 24 drives busy with disk I/O, the entire system only draws about 475 watts. With one of the two power supplies unplugged, the draw drops to 470 watts - a diff of only 5 watts to have redundant power!

Q: Can this chassis sit next to my TV or in my entertainment center?
A: No - the Supermicro 24 bay case is pretty loud, due to the server-class fans. It's meant to sit in a closet, garage, or somewhere else where people aren't sitting. The Norco 4020 case can be made much quieter by slowing down it's fans with a fan controller (Zalman, etc) but it's still debatable whether a 4U rackmount case belongs in the living room or next to a TV.
post #2 of 820
Totally drool worthy !

Least we now know the sole cause of global warming
post #3 of 820
hey odditory, nice to meet ya! i was actually reading your posts over at [h] because someone mentioned your posts there with screenshots of hdtach showing 800MB/s

looking forward to this thread... too late now, but i've got some questions... first, about controllers & speed... more later
post #4 of 820
This reminds me of when I worked at HP on cabinets upon cabinets of storage. At the time, it was all SCSI and 15k RPM 160GB drives (a few years ago), But with a bunch of adaptec SCSI controllers, I was seeing on the order of 1GB/sec sustained b/w. I never thought I'd see that kind of throughput in someone's "home" server so soon .

I am totally drooling over this setup as I am "only" getting my 4th 1TB drive from WD this week.

-Brian
post #5 of 820
Somebody hand me some paper towels...gotta wipe that drool off the floor....

VERY impressive.
post #6 of 820
Quote:
Originally Posted by lanstrom View Post

Totally drool worthy !

Least we now know the sole cause of global warming


Agreed, LOL...

Congratz... just three letters to you... OMG!!! thats beautiful.
post #7 of 820
Why so much RAM? The screenshots seem to confirm that it isn't being used.
post #8 of 820
Thread Starter 
Quote:
Originally Posted by mcnabney View Post

Why so much RAM? The screenshots seem to confirm that it isn't being used.

"Just because" -- I got it cheap and the motherboard has 16 slots and will take 128GB of memory. You can't have "too much" memory on Windows 2003 Server (and soon to be Server 2008).

That said, the transcoding wasn't that memory hungry but I run other stuff on there these days that does have a decent memory footprint (SQL Server 2005 x64, etc).
post #9 of 820
alright i'll ask:

what did it cost? total and $/gb, if you don't mind sharing.

any reason you didn't use the enterprise class (ns) drives instead?

edit:

that is way cool. makes my readynas look like a usb stick.
post #10 of 820
"makes my readynas look like a usb stick. " - lol...priceless!
post #11 of 820
this is an impressive project. Wow, I think its time to upgrade my array...
post #12 of 820
Always nice to see that there are people that are more nuts than I am. Personally I got 8U AIC Rack It has backplanes for 40 drives but physically I can squeeze 6 more but for that I would have to use splitter. I use two Areca 1280ML and Asus striker Extreme motherboard. First 24 slots were populated with 400GB drives. The second controller holds 1tb drives. I add them as needed but now expantion takes more than 24 hours to complete while the whole thing sucks 700W. Soon I will have to upgrade my UPS. Hopefully before the second array reaches its maximum capacity I hope 2tb disks will be out and I will be able to replace 400gb disks.

Attached the picture of the rack on the top in horisontal position is Stacker wich was my first fileserver.


LL
post #13 of 820
odditory: Awesome project. Curious where you got your cases from. I did a quick google and found plenty of places, but newegg doesn't have them. I really don't like buying from other places out of loyalty and general distrust of flyby night shops.


shurik_1: That a homemade rack? If not where did you get it?
post #14 of 820
First I have to say WOW. That is amazing.

My question is, how much heat does that thing put out? Where can you keep it safely? I have often considered my garage, but in the summer it is just to hot.

Wish we have a Storage King so we could tell him to pass you the crown.
post #15 of 820
Quote:
Originally Posted by sosuke3 View Post

shurik_1: That a homemade rack? If not where did you get it?

got it at servercase but the thing is too noisy. PSU fans are at maximum and cannot be slowed down by design. So I have to build something around.

If you are like me line in an appartement consider someting like this
post #16 of 820
Quote:
Originally Posted by Eldron View Post

First I have to say WOW. That is amazing.

My question is, how much heat does that thing put out? Where can you keep it safely? I have often considered my garage, but in the summer it is just to hot.

Wish we have a Storage King so we could tell him to pass you the crown.

mine generates enough heat to keep a room warm in winter but it does not get less than +5 celsius here. Nevertheless drives operate at around 35 all the time.
post #17 of 820
Thread Starter 
Quote:
Originally Posted by shurik_1 View Post

got it at servercase but the thing is too noisy. PSU fans are at maximum and cannot be slowed down by design. So I have to build something around.

If you are like me line in an appartement consider someting like this

One of my projects is to open up the PSU's and slow the fans down with a zalman fan speed controller or similar. At least you can slow the case fans down in the BIOS (of a Supermicro motherboard, anyway).

Quote:
Originally Posted by sosuke3 View Post

odditory: Awesome project. Curious where you got your cases from. I did a quick google and found plenty of places, but newegg doesn't have them. I really don't like buying from other places out of loyalty and general distrust of flyby night shops.

shurik_1: That a homemade rack? If not where did you get it?

Yeah they're kind of hard to find. They only first appeared on the market in January. I got mine from Ingram Micro which is a national distributor of computer equipment and electronics gear. I can get them at cost for $970.00

By the way, SuperMicro also makes a case I recently bought that looks like the 24-bay case cut in half: 2U and 12 drive bays. Still noisy though.
post #18 of 820
Must be nice to have the Web on a Server


So is one of the advantages of having this setup is that if a HDD fails you can just replace it without losing any information?
post #19 of 820
Quote:
Originally Posted by danieloneil01 View Post

Must be nice to have the Web on a Server


So is one of the advantages of having this setup is that if a HDD fails you can just replace it without losing any information?

yes in RAID6 2 drives may fail. I have already replaced 3 drives two of which were dead on arrival and failed to expand. That is why I prefer Seagate for their 5 years warranty.

Actually I would be interested on opinion what happens if backplane fails. if it totally fails on boot it should be still OK as ARECA would detect that as migration and shoud niot initialize RAID but what if it happens when RAID is online???
post #20 of 820
Odditory,

I don't see write speeds posted for your array? How fast are they? Using write-through or write-back cache? I ask because I am only getting 20 MB/sec sustained write speeds from my Highpoint Rocketraid 3520 RAID5 array with 4 1TB Seagate drives using write-through cache (no battery backup). I thought that seemed a bit slow for hardware RAID with the Intel IO340 I/O processor.
post #21 of 820
Quote:


App: MyMovies 2.20 w/ 4875Titles



How long does it take you to settle on a movie to watch?
post #22 of 820
Does Supermicro make a 1U drive bay rack?
post #23 of 820
SC 813T-500, SC 814T and SC815TQ series, all have a 4 port SATA backplane. Once you start moving to a thicker chassis, Supermicro really starts packing the hot swap bays in. Their 4U solution is one of the densest storage solutions (in the 3.5" form factor. I actually saw a chassis some ago that could fit something like 48 2.5" drives, all in a single 4U).
post #24 of 820
How do you propose to handle backups, off-site storage, and disaster recovery? LTO-3 tape drives? Remote data replication?
post #25 of 820
I noticed your set up in MM forum a while back and it was amazing to me at that time. Today when I read it again, it is still amazing to me. I have a 6TB 12 HDD bay RAID 5 server in my garage and compare to this set up, it looks like mine a baby toys.

Btw, what is your next goal? 96TB server?
post #26 of 820
Quote:
Originally Posted by shurik_1 View Post

got it at servercase but the thing is too noisy. PSU fans are at maximum and cannot be slowed down by design. So I have to build something around.

If you are like me line in an appartement consider someting like this

Noise isn't really a problem. I got a 2 bedroom apartment just for my computers....

One thing I do have a problem with is the places you can buy racks from all say they need to be shipped via truck (not UPS/fedex) once you get to a size I would want. I don't have a loading dock so that kind of makes it hard to get one (?) Well the shipping and the price. I really don't understand why < 100$ worth of steel costs 500$.
post #27 of 820
Even if they ship by truck, they will always unload at the front door if requested. You don't necassarily need a loading dock. (I just ordered 20 1U servers that all came in their boxes, and all strapped together on a skid. They unloaded at my front door happily. Of course taking the skid apart and taking each one in separately..was a chore... )You just need a couple of friends..and voila..it's inside.
post #28 of 820
I can hear it already... "Oh my back!"
post #29 of 820
Thread Starter 
Quote:
Originally Posted by shurik_1 View Post



Actually I would be interested on opinion what happens if backplane fails. if it totally fails on boot it should be still OK as ARECA would detect that as migration and shoud niot initialize RAID but what if it happens when RAID is online???

You just force the drives back online. Areca has a function called "recover array".

Quote:
Originally Posted by lifespeed View Post

Odditory,

I don't see write speeds posted for your array? How fast are they? Using write-through or write-back cache? I ask because I am only getting 20 MB/sec sustained write speeds from my Highpoint Rocketraid 3520 RAID5 array with 4 1TB Seagate drives using write-through cache (no battery backup). I thought that seemed a bit slow for hardware RAID with the Intel IO340 I/O processor.

There's something wrong if you're only getting 20mb/s on writes - seriously wrong. I get about 400-500Mb/s, and I'm beginning a migration to Adaptec 52445 cards which have a dualcore XOR (parity calculation) chip that's so fast that write speeds = read speeds - the XOR chip is no longer the bottleneck on latest generation Adaptec and Areca cards.

Quote:
Originally Posted by yngdiego View Post

How do you propose to handle backups, off-site storage, and disaster recovery? LTO-3 tape drives? Remote data replication?

Tapes are useless in the present day, except for transporting offsite where they're more resilient to bumps and vibration than harddisks. With the cost/Gigabyte these days (I've seen 1Tb drives as cheap as $139 recently) its easier to build more arrays to backup to - there are cheaper ways to large arrays than expensive raid controllers (but at the cost of throughput performance) by using software like Flexraid (flexraid.com).

Quote:
Originally Posted by hlkc View Post

I noticed your set up in MM forum a while back and it was amazing to me at that time. Today when I read it again, it is still amazing to me. I have a 6TB 12 HDD bay RAID 5 server in my garage and compare to this set up, it looks like mine a baby toys.

Btw, what is your next goal? 96TB server?

I've stopped posting at MM forums because I'm disappointed with direction Brian's taken with MM in the last 6 months. My next 'goal' is building a large array at the least possible cost, for backup and for curiosity.

Quote:
Originally Posted by sosuke3 View Post

Noise isn't really a problem. I got a 2 bedroom apartment just for my computers....

One thing I do have a problem with is the places you can buy racks from all say they need to be shipped via truck (not UPS/fedex) once you get to a size I would want. I don't have a loading dock so that kind of makes it hard to get one (?) Well the shipping and the price. I really don't understand why < 100$ worth of steel costs 500$.

AIM is a common trucking company that is cost effective and will drop stuff at your front door. I had a 150-lb item delivered the other day and it only cost $40 to ship via AIM. Of course, whoever you're buying from needs to be able to use them as a shipper.
post #30 of 820
odditory

out of curiosity how do you start PSU in the secondary unit? I had a special adapter that came with Stacker to start two PSU simultaneously.

another question why a second RAIS controller? What you think of Areca External RAID Controllers?
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Home Theater Computers
AVS › AVS Forum › Video Components › Home Theater Computers › BUILD LOG: 48 Terabyte Media Server