or Connect
AVS › AVS Forum › Video Components › Home Theater Computers › How long should 20TB of full drives take to initialize flexraid ????? (help me with basic math)
New Posts  All Forums:Forum Nav:

# How long should 20TB of full drives take to initialize flexraid ????? (help me with basic math)

Ok...

So I dropped out three drives (GREEN DRIVES) and I want to initialize my server with my Seagate 3TB and 1 Green WD 2TB. My Green 3TB WD is my Parity drive but this is temp.

So I basically have:

Seagate 3TB#1
Seagate 3TB#2
Seagate 3TB#3
Seagate 3TB#4
Seagate 3TB#5
Seagate 3TB#6
WD 2TB
WD 3TB PRU

Total is 20,000 GB right ?? (20TB) 3000+3000+3000+3000+3000+3000+2000 ( give or take for 1024 thing)

So I am watching it tick by and the lowest I see the Parity Computation process speed reach is about 85MB/s

So that means it's processing data at 85MB per second right ?

So then.. could I divide 20,000 GB by 85MB per second = 235. Does that mean it would take 235 seconds ? That makes no sense to me. Obviously I am an idiot right ?

So then I thought

85MB second = 85x60seconds= 5100MB per minute. That's 5GB right ? So it's doing about 5GB per minute? (Still seems fast right ???)

So that means 20,000 GB divided by 5GB per minute = 4000 minutes. That seems about right ???

so 4000 minutes divided by 60 = 66.6 hours.

That means it's gonna take this pig 66 hours to finish ????

Seriously ????

I mean I have some spare space (at least 200GB per HDD) and the proccess is sometimes faster but with the green drive as parity I think it's limited to a combo of how fast CPU can process parity and how fast the slow green drive can record it.

so I guess 50 hours would be a reasonable expectation ???

holy smokes !!!

### AVS Top Picks

Yes, this is how long it takes... one reason I don't suggest snapshot raid when you have that much data.

The re-parity/resilvering is as fast as your slowest drive.

In your case, it has to read the parity block, then read the data block from 7 DRUs/UoRs, figure out which one is wrong, calculate the missing block, then write it to the correct drive. That is 8 reads, and one write just to recover/get back one block of data.

Parity creation is roughly 1TB an hour. Recovery can be much slower.

With snapshot RAID, you can also run into it's unique un-recoverable mode. Say you lose one drive, and start the recovery process. Another data drive decides to give up the ghost... you now have two drives worth of lost data with no way to get it back... even if you had two PPUs that didn't fail. I guess it's better than total loss, but still discouraging.

It's not much better with a traditional RAID, but at least you know the limits of recovery (number of parity disks).
Quote:
Originally Posted by Puwaha

Yes, this is how long it takes... one reason I don't suggest snapshot raid when you have that much data.

The re-parity/resilvering is as fast as your slowest drive.

In your case, it has to read the parity block, then read the data block from 7 DRUs/UoRs, figure out which one is wrong, calculate the missing block, then write it to the correct drive. That is 8 reads, and one write just to recover/get back one block of data.

Parity creation is roughly 1TB an hour. Recovery can be much slower.

With snapshot RAID, you can also run into it's unique un-recoverable mode. Say you lose one drive, and start the recovery process. Another data drive decides to give up the ghost... you now have two drives worth of lost data with no way to get it back... even if you had two PPUs that didn't fail. I guess it's better than total loss, but still discouraging.

It's not much better with a traditional RAID, but at least you know the limits of recovery (number of parity disks).

Nothing wrong with snapshot with that much data.

Sure, you run the extremely unlikely risk of a second drive "giving up the ghost" during the day or so that you are rebuilding your array.

However, while the chance of you having 2 drives fail within 24-48 hours isn't zero, its very very close. There is also this chance with any other option.
Dividing 20k GB with 85 "MB" is wrong since the units are different. For simplicity, let's say there are 1000GB in a TB. That would mean 85MB is 0.085GB. Now dividing 20kGB by .085GB gives you roughly 235294 seconds which is roughly 66 hours. So yes, it would take 66 hours to compute parity for 20TB. That seems about right. I get 85MB/s parity calculation speed too with my WD Green drives. It took me about 11 hours to compute parity for about 3TB worth of data. And you have roughly 6 times more data than me. 66 hours doesn't sound far off.
Hold on. You are only getting 85 mb/sec after "upgrading"?
Quote:
Originally Posted by Puwaha

With snapshot RAID, you can also run into it's unique un-recoverable mode. Say you lose one drive, and start the recovery process. Another data drive decides to give up the ghost... you now have two drives worth of lost data with no way to get it back... even if you had two PPUs that didn't fail.

If you have snapshot RAID with dual-parity, and you have one data drive fail, then during recovery of the failed data drive, another data drive fails, then you can still recover the data on both failed data drives. That is what dual parity means.
Quote:
Originally Posted by assassin

Hold on. You are only getting 85 mb/sec after "upgrading"?

There must be some other bottleneck then simply the computation of parity. I don't use flexraid, but snapraid is capable of computing parity at more than 100 times that rate on my computer. The hash checksum computation is actually slower than parity computation on my machine, but even that is 1000s of MB/S on my computer:
Code:
``````\$ snapraid -T
Compiled with gcc 4.7.2
Speed test with 4 disk and 262144 buffer size...
memset 28059 [MB/s]
Murmur3 4701 [MB/s]
RAID5 int32x2 11742 [MB/s]
RAID5 mmxx2 21082 [MB/s]
RAID5 sse2x2 30975 [MB/s]
RAID6 int32x2 3518 [MB/s]
RAID6 mmxx2 9017 [MB/s]
RAID6 sse2x2 16683 [MB/s]```
```

That is just a canned test of the computation speed that snapraid does (it does not include drive transfer bottlenecks).

But to give you an idea of what a large snapraid system can do, I have 14 data drives with capacity totaling 42TB, and dual-parity (4TB drives). The last time I did a test to generate all the parity from scratch I had about 33TB of data (not all of the data drives were full). The sync (which generated parity and checksum data) took 13 hours, which comes out to an average speed of 700MB/s of data processed. Obviously the bottleneck was not the speed of computing the parity and checksums, but rather the speed of reading the data drives (and writing to the parity drives).

Since the data drives are read in parallel, the net speed is much higher than that of a single drive. But also, since the speed of a HDD varies by approximately a factor of two from the outer edge to the inner edge, with this many data drives there is a good chance that at least one of the data drives will be reading from the slow region at any given time, so the net data reading speed of a large snapshot RAID system can be approximated by the minimum speed of most of your drives multiplied by the number of data drives.

In my case, I have a mix of 2TB, 3TB, and 4TB drives. The slowest single drive speed is around 60MB/s. So, if any of my 4TB data drives were full, a complete sync would take approximately as long as it takes to read a 4TB drive at 60MB/s, which is about 18.5 hours. Actually, it would probably be a bit faster than that, because the 2TB drives are the slowest, and after about 9 hours they would be completely read and then the slowest would be the 3TB or 4TB drives, so the parity writing speed would slightly increase about halfway through.
Edited by jim2100 - 2/9/13 at 12:40pm
Quote:
Originally Posted by jim2100

There must be some other bottleneck then simply the computation of parity. I don't use flexraid, but snapraid is capable of computing parity at more than 100 times that rate on my computer. The hash checksum computation is actually slower than parity computation on my machine, but even that is 1000s of MB/S on my computer:

The bottleneck is write speed of the disk to be recovered in snapshot RAID schemes. SnapRAID is block-level so it's a bit more efficient than FlexRAID in my opinion. FlexRAID does it's parity on a file-level so it must read entire files to generate parity, or recover in mfusik's case.

A striping RAID scheme helps out with the write-bottleneck.
Quote:
Originally Posted by jim2100

If you have snapshot RAID with dual-parity, and you have one data drive fail, then during recovery of the failed data drive, another data drive fails, then you can still recover the data on both failed data drives. That is what dual parity means.

I hope you are right. I don't see how it's possible to have multiple parity on a file-level without striping. But then again, I don't use it anymore, so I'm not worried about it.
FlexRAID doesn't strip data.
Quote:
Originally Posted by amarshonarbangla

FlexRAID doesn't strip data.

Yeah, I know. Multiple-parity without striping makes my head hurt thinking about the math involved. Props to brahim for making it work... but I won't trust it anymore.

If you have really, really, really static data, then it's fine for most people... but most of us have ever changing data with HTPCs without realizing it. My parity calculations kept taking longer and longer... 12 hours for 5TB was just intolerable for me.
Quote:
Originally Posted by Puwaha

Yeah, I know. Multiple-parity without striping makes my head hurt thinking about the math involved. Props to brahim for making it work... but I won't trust it anymore.

If you have really, really, really static data, then it's fine for most people... but most of us have ever changing data with HTPCs without realizing it. My parity calculations kept taking longer and longer... 12 hours for 5TB was just intolerable for me.

It's not like you add 5TB of data everyday. Once the initial parity is computed, you will probably add a few hundred gigs of data at most on the most demanding days, which won't take long to compute. I don't see why this is an issue at all.
Quote:
Originally Posted by assassin

Hold on. You are only getting 85 mb/sec after "upgrading"?

My Parity is a WD GREEN 3TB. One of my Data drives is a 2TB GREEN I took out of my TIVO HD I scrapped.

I'm doing something strange... I have a Seagate 3TB but it's in my desktop full. I am going to copy over the data to the flexraid. Then swap out the PRU drive for a Seagate.

The 3TB Green is going inside my brothers HTPC in the Silverstone 03 case I did. It's a simple H61 Asrock + G630 build.

It's the only PC hardware he has in his house. No server. No desktop. No laptop. It's an all in one solution for him.

It's odd that I have 18 HDD's larger than 2TB's and I don't have a free drive... lol..

I should have just bought one.

I did remove the Hitachi 2TB and a few WD green 2TB's from my flexraid. I am going to use the 2TB Hitachi in my desktop as my second scratch disk. I'll have 5TB total local storage + SSD's. I plan to Cache it with Intel and a spare agility SSD I have.

The 2TB WD greens I am going to sell on ebay. They fetch about \$75 each.

I'll take the \$ I get and go grab a 3TB Seagate. I think I might run Dual PRU drives.

Thoughts on dual PRU ???
Quote:
Originally Posted by assassin

Hold on. You are only getting 85 mb/sec after "upgrading"?

It was 110+ when I did not have the green drives BTW....
Quote:
Originally Posted by jim2100

There must be some other bottleneck then simply the computation of parity. I don't use flexraid, but snapraid is capable of computing parity at more than 100 times that rate on my computer. The hash checksum computation is actually slower than parity computation on my machine, but even that is 1000s of MB/S on my computer:
Code:
``````\$ snapraid -T
Compiled with gcc 4.7.2
Speed test with 4 disk and 262144 buffer size...
memset 28059 [MB/s]
Murmur3 4701 [MB/s]
RAID5 int32x2 11742 [MB/s]
RAID5 mmxx2 21082 [MB/s]
RAID5 sse2x2 30975 [MB/s]
RAID6 int32x2 3518 [MB/s]
RAID6 mmxx2 9017 [MB/s]
RAID6 sse2x2 16683 [MB/s]```
```

That is just a canned test of the computation speed that snapraid does (it does not include drive transfer bottlenecks).

But to give you an idea of what a large snapraid system can do, I have 14 data drives with capacity totaling 42TB, and dual-parity (4TB drives). The last time I did a test to generate all the parity from scratch I had about 33TB of data (not all of the data drives were full). The sync (which generated parity and checksum data) took 13 hours, which comes out to an average speed of 700MB/s of data processed. Obviously the bottleneck was not the speed of computing the parity and checksums, but rather the speed of reading the data drives (and writing to the parity drives).

Since the data drives are read in parallel, the net speed is much higher than that of a single drive. But also, since the speed of a HDD varies by approximately a factor of two from the outer edge to the inner edge, with this many data drives there is a good chance that at least one of the data drives will be reading from the slow region at any given time, so the net data reading speed of a large snapshot RAID system can be approximated by the minimum speed of most of your drives multiplied by the number of data drives.

In my case, I have a mix of 2TB, 3TB, and 4TB drives. The slowest single drive speed is around 60MB/s. So, if any of my 4TB data drives were full, a complete sync would take approximately as long as it takes to read a 4TB drive at 60MB/s, which is about 18.5 hours. Actually, it would probably be a bit faster than that, because the 2TB drives are the slowest, and after about 11 hours they would be completely read and then the slowest would be the 3TB or 4TB drives, so the parity writing speed would slightly increase about halfway through.

I think your close and accurate on this.

Well done.

My belief is the calculations are much faster than the speed at which data is read or written. The calculations takes a small time, and the majority of the reason it takes so long it because it just pain take a long time to read that much data and write PRU.
Quote:
Originally Posted by Mfusick

It was 110+ when I did not have the green drives BTW....

Something is wrong then. I easily get that speed with my Greens.

I am not going to get into it with you on your "anti-Green" stance but you should know that the newer WD green drives don't have head parking at all. Of course, I have referenced numerous times why I think this is a non-issue for Software RAID but since you think its is still an issue I thought I should mention it to you.
Quote:
The recording head never touches the disk media ensuring significantly less wear to the recording head and media as well as better drive protection in transit.
Quote:
Originally Posted by Puwaha

The bottleneck is write speed of the disk to be recovered in snapshot RAID schemes. SnapRAID is block-level so it's a bit more efficient than FlexRAID in my opinion. FlexRAID does it's parity on a file-level so it must read entire files to generate parity, or recover in mfusik's case.

A striping RAID scheme helps out with the write-bottleneck.

It is going to take a long time to read 20TB of data no matter how you slice it.

What do you use ? ZFS ???
Quote:
Originally Posted by amarshonarbangla

It's not like you add 5TB of data everyday. Once the initial parity is computed, you will probably add a few hundred gigs of data at most on the most demanding days, which won't take long to compute. I don't see why this is an issue at all.

lol.

Speak for yourself.

I often do my organizing locally on my desktop. When my scratch disk is near full I empty it to the appropriate storage area on the server.

I might rip 10 blurays or process a few weeks of TV shows before I add them to the server. So I certainly copy or paste more than a a few GB. I'll do a 1TB or more on many occasions.

At 25+ per Bluray... and about 1GB per TV show it adds up quick. I have 2 scratch disks on my desktop I organize my media on these. Run Mediamaster- rename, do the album art and meta data thing. Once the folder is 100% done and ready only then I move it to my server.

That way I know everything on my server is 100%.

It's not allowed on the server unless it's complete and worthy of long term storage. Only about 3TB of my server is not media. Most of it is media.

I have the typical pictures, programs... and such too. But it's not more than a few TB's.
Quote:
Originally Posted by Mfusick

lol.

Speak for yourself.

I often do my organizing locally on my desktop. When my scratch disk is near full I empty it to the appropriate storage area on the server.

I might rip 10 blurays or process a few weeks of TV shows before I add them to the server. So I certainly copy or paste more than a a few GB. I'll do a 1TB or more on many occasions.

At 25+ per Bluray... and about 1GB per TV show it adds up quick. I have 2 scratch disks on my desktop I organize my media on these. Run Mediamaster- rename, do the album art and meta data thing. Once the folder is 100% done and ready only then I move it to my server.

That way I know everything on my server is 100%.

It's not allowed on the server unless it's complete and worthy of long term storage. Only about 3TB of my server is not media. Most of it is media.

I have the typical pictures, programs... and such too. But it's not more than a few TB's.

Well given I write all my media directly to the pool, it really isn't more than a couple hundred gigs in a day, at most. But yea, usage varies from person to person
Quote:
Originally Posted by assassin

Something is wrong then. I easily get that speed with my Greens.

I am not going to get into it with you on your "anti-Green" stance but you should know that the newer WD green drives don't have head parking at all. Of course, I have referenced numerous times why I think this is a non-issue for Software RAID but since you think its is still an issue I thought I should mention it to you.

I have 2 different 3TB WD greens. They are 30EZRX models.

I have 4 different 2TB GREENS. BOTH 20EARS and 20EARX. A mix of WD GREENS.

Gotta love HOT SWAP^
Also,

I have 8 Seagate 3TB 7200.14's
I have a Hitachi 2TB
I have some Samsungs 1TB's in RAID 0

I also have a few 750GB and 1TB's ...

I'm telling you consistently on all 6 of my WD green drives they all slow down when they are full. Empty and fresh formatted they might produce a respectable 100MB/sec but that fades fast. Once you fill them up your getting less than 85MB's.
I am not talking about 1 or two. I am talking about all 6 of mine. 100% of the time.

They are formatted fresh the same way I did the others. Windows default NTFS.

In contrast.. the Seagates deliver 120MB+ on many occasions locally and also consistently delivery faster than 85MB over my network. The WD GREENS are slower than my network and the Seagates are faster.

I don't see it as a big deal since the network speed varies from 85-110MB anyways real world. They are all pretty close so if one bottle neck is removed another appears pretty quickly.

Fill up your green so it's full and tell me what you get ?? I would be very curious.

My Seagates maintain the superior speeds even when near full. There is no difference in heat or noise. Energy profile is acceptable and on par with the Greens.

I just prefer them mostly because the cost per GB is better, but the performance boost is a nice bonus.

Nothing wrong with a green drive but I have decided from personal experience I do not like them and prefer others. I'd buy if they were \$20 cheaper but since they are not I can not recommend them.

I have yet to hear you say once that provided the greens are the same cost or more at the time of availability they are probably a poor choice. Yet I have said many times if they were cheaper they would be a fine choice.

And if the head parking feature is removed on newer models- That just tells me that it was an issue. Why else would it be removed?
Quote:
Originally Posted by Mfusick

It was 110+ when I did not have the green drives BTW....

I am pretty sure your slowest drive becomes the bottle neck.

The parity process requires data to be read from all your drives so the speed your slowest HDD performs at is the total speed your seeing.

That is why my 2TB green WD limited my speed to 80MB/sec.

When I remove that drive and replace I'll be curious to see what the speed is.

When I used only 3TB seagates the process completed with an average speed of 110MB/sec.

an empty drive will hit 135MB/sec I see often. But real world its a bit slower.
Would you be interested in selling me your 3TB green drive
Quote:
Originally Posted by amarshonarbangla

Well given I write all my media directly to the pool, it really isn't more than a couple hundred gigs in a day, at most. But yea, usage varies from person to person

There is a couple reasons I like it my way.

First, I enjoy the superior performance of the locally installed faster HDD's. I use a RAID0 Array and an SSD cache speed boosted HDD (7200rpm) so the performance I see is greater than my network or server would show me.

Second,

I find the performance of MediaCenterMaster to leave much to be desired over LAN. It performs much better on a smaller sized library that is locally installed on a fast HDD. I get lock ups, Freeze and crashes when I try to use it on my server over LAN on a big collection. Not sure why. I am going to be trouble shooting this issue with the developer in the next week or so...

Third,

I find that running mediacentermaster or changing metadata on flexraid often leaves a mess when you break down your array. You might have multiple folders for the same movies spread across different drives. Some metadata is on one drive. Some others on another... That is a mess.

In contrast- I find if I do the renaming and folder management locally and then copy and paste only a 100% complete movie or TV show folder to the server then the entire folder and all it's contents ends up on one drive. I like that better.
When you go back and modify after the fact if your drive is more full flexraid chooses a less full drive to hold the additional stuff. I don't like that. If I break down my array I want each folder on each HDD to be complete.

4th,

Basically that^ I like having only complete folders on my server and only one on one drive. If I do it 100% complete on the desktop and scratch disc then move it to the server for storage then I know everything on my server is 100% all the time. If you have incomplete stuff on your server- then you use a metascraper and add the missing info, rename it- move the video file into a folder etc... you can end up with a mess when you break down your array.

I point my HTPCs to the different libraries on my server so I know those libraries are always 100% complete. I use stuff like HD movies, TV shows, Non HD movies (dvd) Disney etc..

I also have a folder shared on my desktop in the root of the scratch disc that is called Desktop New Media Finalized. Once I run mediamaster on the the video file and it's 100% complete I place it into that folder. Since it's a "LIBRARY" on my desktop and in the root of the HDD the move does not require a copy paste- I have it set up so I can just drag and drop it and relocate it. All finalized moves go there.

That way all the newest stuff is there before it hits the server. I point my HTPC to this folder as "NEWEST MEDIA" So if I want to watch something very new that is where I get it.

It's a complicated and manually managed way to have a new media section on my HTPC. But I find it works well because I often find myself wanting to watch new stuff- and it's easier to find the new stuff I want to watch since this library is small.

I move everything to the server as I find time. This library is typically about 1TB.. not much more ever. It's basically a running total of the newest media I have and as it fills up or new media replaces old media it gets moves.

It's like "being on deck" for my media before it goes to the server.

I know my ways are not for everyone but it's just how I do it.

I would love to hear a solution to some of the issues I find or if someone has a better way of doing this.

I find the work that goes into managing a big and ever growing library is a lot of work.

My wife is about 80% HTPC now.. and less cable. I even have my parents using HTPC. (I do most of the work and they don't do TV)
Quote:
Originally Posted by amarshonarbangla

Would you be interested in selling me your 3TB green drive

One is going in my brothers HTPC I built him:

It is going to go right there ^^^

I think a 3TB WD green would be a good HDD for him in that little machine.

I have 2 of them.

I would sell you the second. (the one in the picture)

But I am about a week away... I need to copy some data and change a few things in preparation for it's removal.

If you don't mind waiting a week sure. It's not old at all. The original failed and this is the replacement I just got from WD.
Quote:
Originally Posted by amarshonarbangla

It's not like you add 5TB of data everyday. Once the initial parity is computed, you will probably add a few hundred gigs of data at most on the most demanding days, which won't take long to compute. I don't see why this is an issue at all.

My usage pattern has a lot more change than most others probably. It's one reason why I outgrew FlexRAID. Perfomance was a primary issue as well. On a fresh FlexRAID install, everything was fine, but over the course of 6-months to a year, parity would take longer and longer to calculate.
Quote:
Originally Posted by Mfusick

It is going to take a long time to read 20TB of data no matter how you slice it.

What do you use ? ZFS ???

I now use ZFS, yes. I had a bad hard drive when I built the pool (I thought it was ok, but ZFS found out that it was not... FlexRAID never told me!), and when I replaced it in my pool, it resilvered 5TB of data in 2.5 hours. It probably would have been faster, but I still have some legacy green drives in the pool. If I had your 20TB, it would have taken me about 10 hours to replace a HD. Not bad, I think. I'm blown away by the performance of OpenIndiana. It serves as the main data storage in the house, as well as my VMWare storage back-end.
Quote:
Originally Posted by Puwaha

I hope you are right. I don't see how it's possible to have multiple parity on a file-level without striping. But then again, I don't use it anymore, so I'm not worried about it.

Of course I am right, this isn't rocket science. Striping has nothing to do with the parity math. You must be confused about something, but I cannot tell what it is.

Regardless of how the data is organized on the drives, the math is exactly the same. You take N data blocks and calculate P parity blocks from the N data blocks. How the parity blocks are distributed over the drives makes no difference at all to the math used to calculate the parity.
Quote:
Originally Posted by amarshonarbangla

Would you be interested in selling me your 3TB green drive

https://westerndigital.secure.force.com/WarrantyCheck?lang=en

Warranty looks good till 2014 btw... I just checked.
Quote:
Originally Posted by Puwaha

I now use ZFS, yes. I had a bad hard drive when I built the pool (I thought it was ok, but ZFS found out that it was not... FlexRAID never told me!), and when I replaced it in my pool, it resilvered 5TB of data in 2.5 hours. It probably would have been faster, but I still have some legacy green drives in the pool. If I had your 20TB, it would have taken me about 10 hours to replace a HD. Not bad, I think. I'm blown away by the performance of OpenIndiana. It serves as the main data storage in the house, as well as my VMWare storage back-end.

I just need to learn a little more first.

For now Flexraid suits me great.
What should I set WHS2011 power options to be ?

Should I turn of HDD "NEVER" or should I turn off after say 30 min ????