Flexraid help deciding - Page 2

I have not seen scattering. What it appears to do from my experience is say I have a 2TB drive with 1.5TB taken up by movies, and then another couple drives out there with varying amounts of space. It seems to fill up that 2TB drive when I copy something into my My Movies folder. Once it is filled, it creates a My Movies folder on another drive and begins copying movies there until that one fills up. From what I can tell, it does not do any type of load balancing between drives (I add two 2TB drives a while ago and they are still largely empty, despite my other 4 being almost completely full each).

+1
Agree

If you want to control the pool so much.. your better off just deleting a drive from the pool or not adding it in the first place.

That's not really the point of flexraid.

however... I am pretty sure if it's already on a drive it will stay there. Its only new stuff you add that you don't control...

right

I'm just quoting FlexRaid wiki here but it seems there are different options in FlexRaid for how this is dealt with:

Storage Pool Merge Modes

FlexRAID Storage Pool supports several merge modes:
- an automatic mode with balanced space across drives
- an automatic mode reducing folder split across drives
- a manual mode where you explicitely configure the merge

Auto Merge with Balanced Space Priority

With this mode, the configuration is automatic and there is no configuration to deal with. All folders and files already on the drives will be available in the pool, and when you add folders and files to the pool, they will be written to the different drives depending on their available space.

This mode is very similar to what you are used to with WHS with the difference being in that it balances the disk space so that the drives are used evenly.

Auto Merge with Minimized Folder Split Priority

As above, the configuration is automatic in this mode and there is no configuration to deal with. Again, all folders and files already on the drives will be shown in the pool, and when you add folders and files to the pool, they will be written to the different drives in order to minimize the folder split: rather than balancing the data based on free space, this mode tries to keep folder splits to a minimum.

This mode will fill up each drive one by one, and as such can cause certain drives to be used more than others. But it is more energy efficient.

Explicit Merge

This is the mode you are all familiar with from older versions of FlexRAID. This mode allows for a lot of flexibilities and really intricate configurations.

It allows you to treat your source data as a database, and to create Views into that data (no need to mess with symbolic/hard links).

Additionally, it provides the greatest energy saving feature.

I am too looking to expand the storage on the server. Bear with me, I am running Windows 2000 Server on a Pentium II 233 MHz, running RAID5.

Perhaps my mind is stuck in a traditional RAID where redundancy is achieved by writing in 2 locaitons.

If, for the simplicity of mathematics, I have 5x 3Tb drives, and I dedicate 4 of them to the pool, and 1 of them to the parity, then my pool should be 12 Tb, correct?

How is the parity achieved on the single 3 Tb drive? I just can't get over the notion that 12 Tb of data can squeezed into 3 Tb of space...

Should 1 of the 4 pooled drives fail, and I replace it with a blank, will FlexRAID rebuild the data in the pool using the 3 Tb parity drive?

Will the array be accessible during the rebuild (like RAID 5)?

Sorry for all the questions.

I would like to utilize my older IDE drives, which are all sub 500 Gb, which will push the drive count over 20, and UnRaid requires the "pro" license for the high drive count.

Just one correction to a previous answer. You CAN set a certain percentage. It's under the settings and preferences and called "Reserve". It is worth playing with, as it will determine when files in a folder are "split" between drives. It can also make a difference as to which drive is the largest (think parity drive) when dealing with drives that are supposedly the same size but from different manufacturers.

That's the default mode. You can change it to auto-space priority to "balance the space available on the drives".

That's what makes UnRaid and Flexraid proprietary systems, and they're not telling exactly how they achieve "parity without striping".

Yes, it will rebuild the drive from the parity drive. I've done it on a small scale as a test. I believe, but am not absolutely sure, that the array is available during rebuild, but the drive that is being rebuilt is not available. Could be wrong on that however.

Good replies BomberJim

lol

Parity is a pretty cool concept.

Parity explained:
http://www.youtube.com/watch?v=clvTWV1bgdI

I will try at the above...

But I need to break it down so my ADD mind works properly.

It's definitely a different concept that you can back up 15TB of data with 3TB of space.

(5) 3TB drives with 4 as DATA drives (called DRU) and the remaining 3TB drive as a PPU (Parity drive) [back up]

So 4 data, one parity. Then you would have a pool size of 12TB. Yes. 4x3=12. 3TB Parity would provide backup if one of those drives went down. You could actually remove one of those drives and replace it with a blank 3TB drive and re-write the data back to it. Pretty cool if you ask me.

Parity keeps track of the total - it's math.

remember that all digital information is stored as either a 0 or a 1. it's never anything different.

so basically it reads each unit across all the drives and records the sum on parity drive.

Look at this:



That is a good example.


It records the totals across your drives and stores the answer on the parity drive. That is why the parity drive must be as large as your largest drive so it can hold that many parity totals.

If you subtracted one of those data drives you could figure out what is missing or what digit the missing drive had. Just look at the digits you know... and the answer and you can figure out what it was. That is basically how flexraid works.
You can back up more than 5 drives. I use it on 8. But as you increase drive counts you increase the chance of simultaneous drive failures. That's a problem.

Last-

Yes.

Your "pool" is always available. Flexraid is real time. Once you set it up- your drive is always available unless your doing a rebuild. You can get an email each day on the status if you want to know it's ok. It can email you your parity is up to date and correct if you need to know or want to be sure. I don't bother.
also,
Flexraid will allow you to use your older drives. You can then replace them with larger drives in future. I am doing this. replacing a 500GB drive with a 3TB gains you 2.5TB of space without needing another SATA port or SATA power. It's the way to do it when you find $$$ and good deals on large drives. Plus- it's good to replace older drives before they fail. it's just easier.

Yep, it's math, and the first linked video is another good explanation. You're not backing up (or protecting) files, you're playing with digital data - 1's and 0's. Where the magic comes in is when you add a second parity drive to guard against two data drive failures. The calculations are now not so straight forward. Flexraid allows an unlimited number or parity drives.

I am thinking of adding a second Parity drive.

Do you do this?

What is the recommendation ?

unRAID parity is pretty well explained. It's only proprietary because to achieve what they did, they had to create their own solution that was on a kernel level.

FlexRAID drives will not be available during a rebuild for realtime pooled arrays (snapshot arrays it might be - haven't tried). unRAID can simulate failed drives the same as a hardware RAID controller does. I pulled a drive out of my array and was still able to read the contents that were on that drive from the server. The drive had many read/write errors and so accessing it while it was still in the server would occasionally hit points that slowed it down to a crawl or stall the server. Taking the drive out allowed the drive to be read at the speed of parity creation. Once a new drive was placed in, the contents were re-created. Accesssing the array is possible during this time, but slows down the rebuild process so is not really recommended as more time spent building increases the chances of a second failure.

Are you using realtime parity protection? Last I remember, you can't do dual parity in that mode.

Apologies for going off on a bit of an tangent here but I'm deliberating what OS for Flexraid. I had assumed Win 7 is all I need but hear some are using WHS. Can anyone tell me what is the main reasons for using WHS over plain old Win 7?

Thanks

It's cheaper than Win 7, it's a server OS, and it's built in backup of client computers (and itself) is very nice as well. If your designing a multi-purpose machine Win 7 is a good choice, if it's just a server then WHS 2011 should at least be considered.

I am in the exact same situation as you and wondered about this exact situation. I would rather not have music and pictures which are small files in comparison to a full bitrate MKV spanning accross multiple drives in the pool. Like you said, if for some reason I wanted to break up the pool, I'd rather not deal with hunting through how ever many drives you have to find all the music or picture files.

i was thinking and I'm hoping Flexraid can do this: Keep your pics and music on a larger unpooled drive but still have the parity drive "see" this drive as well when it calculates the parity? Can Flexraid do this?

In other words... Can Flexraid parity the pooled drives AND additional un-pooled drives with a single parity drive?

Whs is much cheaper and better suited for a pure server.

You guys are over concerned with small files spanning drives.

It's a total non issue.

Lets say you have a music folder and you add a few mp3 files.. They go into that folder on that drive.

It's only when that drive fills up that it would get stored to another drive.

And not all drives. One drive. Inside a folder also named music.

If you broke down your array you would see you have a music folder on two different drives. Only reason it's on two drives is cause the first one was full.

I think everyone is over reacting to the whole "spanned across multiple drives small files" thing. It's not an issue.

When you break up your pool you don't have random small files all over the place.

The solution is of coarse in your pool have a small amount of top level drives with everything inside that.

Example: folder for video. All movies and TV shoes inside it. I have ten folders or categories in mine. HD not HD Bluray DVD TV shows Disney etc.. Those are my libraries I point my mediabrowser to. You have to set it up this way anyway if you want it to work properly.

Then on my pool I have folder called music: anything has to do with music inside. In folders. Organized.

Then I have folder called programs: anything has to do with programs inside

Then I have folder called documents: anything appropriate inside.

Then I have folder called misc.

That's it.

So of I break my array up then everything is still organized in just a few folders.

It's not like my music is all over the place.

It's all inside a music folder. Perhaps on two drives but that's only because my music drive filled up. Otherwise Flexraid would keep filling the original drives this is the default.

If I had 6 DRU's, and 1 PPU, and I had two data drives fail, or 3, what would happen? Would I loose one of the drives, or could that PPU rebuild both?
If I had two PPU's, does that protect the data in case one of the PPU's fails during a rebuild?




Thanks!

Thanks ncarty97 & bomberjim too

Depends.

You can use more than one parity drive which can provide additional security for a much more unlikely event. Flexraid can guard you against two drives failing- if that is what your asking.

I just think it's unreasonable to think your going to have more than 2 drives fail at the exact same time. Unless of a house fire or flood or EMP terrorist attack it seems unlikely a normal level event would happen to cause that.

If your worried about that- CRASHPLAN offers unlimited backup with remote cloud for a very affordable price.

You could just back up you entire server offline. That's a better idea for someone who is ultra data loss and data security concerned.

I am cool with just simple double parirty. I plan to use only two parity drives all the way up to 30TB pool. I think it's reasonable to go about 8 drives per parity drive before you need to add more parity. You can of coarse go double parity on 6 drives in a pool for extra security on simultaneous drive failure.

So one parity drive can protect against 2 or more drives failing at once?
Having multiple parity drives just protects the data in case the parity drive fails?
How many drives can you have on 1 parity?