Mfusick's How to build an affordable 30TB Flexraid media server: Information Requested.! - Page 82 - AVS Forum
Forum Jump: 
Reply
 
Thread Tools
post #2431 of 3764 Old 01-03-2014, 06:07 PM
AVS Addicted Member
 
assassin's Avatar
 
Join Date: Jul 2004
Posts: 12,894
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 205
Quote:
Originally Posted by Mfusick View Post

You really are thick sometimes. And stubborn. And you do exactly what you accuse me of doing. Assume everyone is in your shoes too.

When did I say NEED ? I never did. I just said "ideal" and "less ideal" Your waging some strange war just trying to pick a fight or argue for whatever reason on one of the stupidest subjects. You see everything as so black and white. I just think that smaller than 4TB drives and cheap fast 7200rpm drives are ideal for a flexraid server. If I was going to design one from scratch that is what I would choose because it works the best, and costs the least. I never said nothing else would work or that you NEED this.

You imagine things just to argue against them.

What? More utter nonsense.

Look just because you use words like ideal when referring to how YOU use a server or htpc does not at all make it a correct statement. Its just your opinion. 1 data point from 1 user. And unless you have tried each and every other option that is being discussed (you haven't) you really have no background to be discussing the "ideal" options at all. You can post your experiences but trying to point out how other's experiences are "not ideal" is just wrong and ignorant. You constantly want to try to convince people that the ideal way for you should be the ideal way for everyone.

This thread is case in point. How is what I am arguing "black and white"? Because it differs form your "ideal" viewpoint of the world? What an absolute ridiculous and narrow minded comment.

When you say something is "ideal" it implies that anything less is "not ideal". Again, this is rubbish.

So allow me to revise my statement since it appears you don't like my use of the term "need"...
  • 7200 RPM drives are not IDEAL
  • Completing the parity as quickly as possible for fear that you are somehow going to corrupt your machine if you accidentally press a button during verify/validate/update is not IDEAL
  • It isn't IDEAL to have a specific size of drive --- use your old 2TB, 3TB or (*gasp*) even 4TB. Use whatever you have that makes sense to you financially and makes the server work for how YOU will be using it.
  • There is no such thing as parity performance --- its a parity drive. It sits there and holds redundancy data. Having a "faster" parity drive will not somehow improve your server experience. In the end it will playback your movies exactly the same each and every time.
  • You don't even need to know when your server is computing an update or other task.
  • What MFusick decribes as his "ideal" server software is actually ideal for anything but a HTPC (or home PC user) for that matter. Most of us will not be constantly changing our media every hour of every day (like a business would for instance). In almost everyone but MFusick's situations snap shot is ideal as it is simple and you can use simple and inexpensive hardware to achieve it. Some people only perform an update manually after they update their server as it is that infrequent. This is also a correct way of doing it. Again, there is *gasp* more than one way to build and use a server or HTPC.
  • Bonus: It is not IDEAL to use small and "faster" drives based on benchmarks

And finally my favorite post of the week...
http://www.avsforum.com/t/1505076/htpc-front-end-shootout-show-us-your-eye-candy/120#post_24155797
assassin is offline  
Sponsored Links
Advertisement
 
post #2432 of 3764 Old 01-03-2014, 06:18 PM
AVS Special Member
 
kapone's Avatar
 
Join Date: Jan 2003
Posts: 4,313
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 9 Post(s)
Liked: 39
I get a headache reading this thread... frown.gif
Mfusick likes this.
kapone is offline  
post #2433 of 3764 Old 01-03-2014, 06:18 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,047
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 124 Post(s)
Liked: 255
Quote:
Originally Posted by Mfusick View Post

No. I don't think so. Most of the stuff I do is not that Dynamic. Moving data, adding or deleting is perfectly fine with snapshot.

I guess I need clarification on what you mean by "perfectly fine." I know you can perform those tasks without flexraid dying or anything like that, but anything you move (from drive to drive) add or delete, will require a parity update to keep everything in sync, no? The big downside to snapshot is that you lose redundancy when your parity is out of sync, right? Isn't that why you try and run a parity update on a fairly regular basis, and after large write operations?

Quote:
You can do it all you want. Even during parity computation processes. Changing data is the bad part. I do mostly adding, deleting, moving. Very little changing.

How is adding or deleting a file different from changing? The bits on the drive change regardless, right? The parity is calculated from the bits on the drive, right? It still means your parity is now out of date, right? I fail to see the difference.

Surely it doesn't just generate parity based on filenames. That's the only reason why I can see the system behaving differently with modified data, vs new/deleted/moved data.

Quote:
BTW- you really need to get your post count up so AVS grants you permission to edit, delete your posts. Your Ninja edits and post after a post are hard to keep up with tongue.gif

I can edit posts. As far as I know, I've always been able to. I'm just not going to go back and edit every post I make because you edited yours while I was writing my initial response. You've quoted me in 4 posts today yet I have nine emails notifying me that you quoted me. Perhaps you should gather your thoughts and put them all in one post the first time wink.gif

If I had a signature, this is where it would be.
ajhieb is online now  
post #2434 of 3764 Old 01-03-2014, 06:18 PM - Thread Starter
AVS Addicted Member
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 21,724
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 79 Post(s)
Liked: 683
Quote:
Originally Posted by assassin View Post


When you say something is "ideal" it implies that anything less is "not ideal".

I was right. You don't get it.

Here is a clue since you need it.

Try "less ideal" instead of "not ideal". Big difference.

Again, you are making stuff up and projecting it onto me just to argue with me. It's a common occurrence you do and it's annoying. You are more interested in trying to prove me wrong than you are interested in trying to help the guy asking the question. It means more to you to show me "wrong" than it does to answer his post on the subject with something that's useful. By my observation you spend about 50% of you time on AVS going this behavior. The other 50% is spent promoting or discussing your products or services.

When you make something so black and white when it is not- it makes it easy to argue. That seems your intention. Just because I believe smaller faster hard drives are better for a flexraid server doesn't mean I am saying you NEED them or anything else is NOT IDEAL I am not understanding how you make this jump and conclusion. If you go back and read this all again you will see where I told him he could do it, and after making him aware of why you might or might not I even recommended it!
Quote:
Originally Posted by kapone View Post

I get a headache reading this thread... frown.gif

I am sorry. I wish Assassin would stay out of this thread honestly. AVS was better when he was gone. Much less arguing over stupid things.

-

"Too much is almost enough. Anything in life worth doing is worth overdoing. Moderation is for cowards."
Mfusick is online now  
post #2435 of 3764 Old 01-03-2014, 06:22 PM
AVS Addicted Member
 
assassin's Avatar
 
Join Date: Jul 2004
Posts: 12,894
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 205
Quote:
Originally Posted by ajhieb View Post



I guess I need clarification on what you mean by "perfectly fine." I know you can perform those tasks without flexraid dying or anything like that, but anything you move (from drive to drive) add or delete, will require a parity update to keep everything in sync, no? The big downside to snapshot is that you lose redundancy when your parity is out of sync, right? Isn't that why you try and run a parity update on a fairly regular basis, and after large write operations?


How is adding or deleting a file different from changing? The bits on the drive change regardless, right? The parity is calculated from the bits on the drive, right? It still means your parity is now out of date, right? I fail to see the difference.

Surely it doesn't just generate parity based on filenames. That's the only reason why I can see the system behaving differently with modified data, vs new/deleted/moved data.
I can edit posts. As far as I know, I've always been able to. I'm just not going to go back and edit every post I make because you edited yours while I was writing my initial response. You've quoted me in 4 posts today yet I have nine emails notifying me that you quoted me. Perhaps you should gather your thoughts and put them all in one post the first time wink.gif

You are correct ajhieb. Everytime you make even a single keystroke or change data you are altering what is stored on your DRU drives. This in turn by definition changes what needs to be written to the Parity drive. It doesn't matter if it is changed, added or deleted. Its the fact that the data has been altered from the previous parity redundancy.

That why I suggested that if he is truly using his server in this manner then he has most definitely (and likely unknowingly) altered some data during the update, verify or validate process. I also agree with you that anyone that is constantly changing data (again ---- more like a business) is much more suited for real time than snapshot.
assassin is offline  
post #2436 of 3764 Old 01-03-2014, 06:25 PM
AVS Addicted Member
 
assassin's Avatar
 
Join Date: Jul 2004
Posts: 12,894
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 205
Quote:
Originally Posted by Mfusick View Post

I was right. You don't get it.

Here is a clue since you need it.

Try "less ideal" instead of "not ideal". Big difference.

Again, you are making stuff up and projecting it onto me just to argue with me. It's a common occurrence you do and it's annoying. You are more interested in trying to prove me wrong than you are interested in trying to help the guy asking the question. It means more to you to show me "wrong" than it does to answer his post on the subject with something that's useful. By my observation you spend about 50% of you time on AVS going this behavior. The other 50% is spent promoting or discussing your products or services.

When you make something so black and white when it is not- it makes it easy to argue. That seems your intention. Just because I believe smaller faster hard drives are better for a flexraid server doesn't mean I am saying you NEED them or anything else is NOT IDEAL I am not understanding how you make this jump and conclusion. If you go back and read this all again you will see where I told him he could do it, and after making him aware of why you might or might not I even recommended it!
I am sorry. I wish Assassin would stay out of this thread honestly. AVS was better when he was gone. Much less arguing over stupid things.

What you see as "stupid" I see as "properly informing". Some of your information is just plain wrong as I have pointed out.

Great point on wishing me gone. I know there are plenty that disagree with that statement and prefer a voice of reason.
assassin is offline  
post #2437 of 3764 Old 01-03-2014, 06:27 PM - Thread Starter
AVS Addicted Member
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 21,724
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 79 Post(s)
Liked: 683
Quote:
Originally Posted by ajhieb View Post



I guess I need clarification on what you mean by "perfectly fine." I know you can perform those tasks without flexraid dying or anything like that, but anything you move (from drive to drive) add or delete, will require a parity update to keep everything in sync, no? The big downside to snapshot is that you lose redundancy when your parity is out of sync, right? Isn't that why you try and run a parity update on a fairly regular basis, and after large write operations?

My understanding it that if you add or delete data it will not throw off the integrity of your parity. So by perfectly fine I mean you can either add or delete data and not harm the integrity of your parity computation process.

If you move data, that is basically adding or deleting so I understand this to be ok too with snapshot.

Parity update on my server is set to update every 24 hours so I don't bother manually running it. It is will happen soon enough automatically. It updates at night at 1am. If I don't change much it won't take long.
Quote:
Originally Posted by ajhieb View Post


How is adding or deleting a file different from changing? The bits on the drive change regardless, right? The parity is calculated from the bits on the drive, right? It still means your parity is now out of date, right? I fail to see the difference.

Surely it doesn't just generate parity based on filenames. That's the only reason why I can see the system behaving differently with modified data, vs new/deleted/moved data.
I can edit posts. As far as I know, I've always been able to. I'm just not going to go back and edit every post I make because you edited yours while I was writing my initial response. You've quoted me in 4 posts today yet I have nine emails notifying me that you quoted me. Perhaps you should gather your thoughts and put them all in one post the first time wink.gif

I don't think adding or deleting a file will upset integrity. I think changing it will. Changing would be like opening up a MS word document and editing the text and saving it. But keeping the name the same. Adding would be like pasting a movie from your desktop to your server. Deleteing would be deleting it from your server. They are different.

Adding or deleting will not disrupt parity or throw an error. Changing it would. And if you do lots of changing you would want real time. If you data is stagnant, (like media storage) and you do mostly adding or deleting then snapshot is a good choice. That's my understanding anyways. I could be wrong, but I do not think that I am.

Quote:
Originally Posted by assassin View Post

You are correct ajhieb. Everytime you make even a single keystroke or change data you are altering what is stored on your DRU drives. This in turn by definition changes what needs to be written to the Parity drive. It doesn't matter if it is changed, added or deleted. Its the fact that the data has been altered from the previous parity redundancy.

That why I suggested that if he is truly using his server in this manner then he has most definitely (and likely unknowingly) altered some data during the update, verify or validate process. I also agree with you that anyone that is constantly changing data (again ---- more like a business) is much more suited for real time than snapshot.

I don't think you are right on this. I am talking about doing while you are running parity. I think you can add or delete data while you run parity processes, but you can't change data. If you change it will upset the integrity or cause a failure. Change is different than add or delete I think.

-

"Too much is almost enough. Anything in life worth doing is worth overdoing. Moderation is for cowards."
Mfusick is online now  
post #2438 of 3764 Old 01-03-2014, 06:27 PM
AVS Addicted Member
 
assassin's Avatar
 
Join Date: Jul 2004
Posts: 12,894
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 205
Quote:
Originally Posted by Mfusick View Post

My understanding it that if you add or delete data it will not throw off the integrity of your parity. So by perfectly fine I mean you can either add or delete data and not harm the integrity of your parity computation process.

If you move data, that is basically adding or deleting so I understand this to be ok too with snapshot.

Parity update on my server is set to update every 24 hours so I don't bother manually running it. It is will happen soon enough automatically. It updates at night at 1am. If I don't change much it won't take long.

This is not correct at all. You need to go back and read how a parity drive works before offering up the "ideal" way on how to use it.
assassin is offline  
post #2439 of 3764 Old 01-03-2014, 06:33 PM
AVS Addicted Member
 
assassin's Avatar
 
Join Date: Jul 2004
Posts: 12,894
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 205
Quote:
Originally Posted by Mfusick View Post

I don't think you are right on this. I am talking about doing while you are running parity. I think you can add or delete data while you run parity processes, but you can't change data. If you change it will upset the integrity or cause a failure. Change is different than add or delete I think.

Snapshot literally takes a "snapshot" of the data on your data drives. Since pooling in flexraid determines where data is stored when you move something you are altering what is on the parity drive after an update. How would deleting something or adding something not be changing what would be on the parity drive?

In effect there are two "copies" of data at all times on the server. One is one the pooled DRU drives (this is usually the most current). There is also the "saved" or "redudant" data on the parity drive. If you have not altered anything at all on your server (no adds, no deletes, no modifications) then you will have the exact same data on the DRU drives and parity drives after an update. However, if you alter even one piece of data on the DRU drives (delete even one jpeg, add even one Mp3, move even one movie) then you have changed what is on the DRU drives. Now what you have on the DRU drives and the parity drives does NOT match. In order to get it to match up again (i.e. make it redudant ---- which is the whole point on FlexRaid) you have to run the Update again. Until then you will have the "new" data on the DRU drives and the old data on the Parity drive.

That's why if you add, delete or change something during an update you can just update it again when you are finished. That's also while real time mode is a much better mode for someone like yourself that is "constantly moving data" around as it protects your parity (redundant) data in real time.
assassin is offline  
post #2440 of 3764 Old 01-03-2014, 06:39 PM - Thread Starter
AVS Addicted Member
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 21,724
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 79 Post(s)
Liked: 683
If you make changes you should re-run update I agree with. But if it happens automatically in less then 24 hours you might just wait it out is my approach. I don't re-run updates after making changes. I see no point. But in the longer validate or verify processes I think you can't change data or it will throw an error or mess with the integrity. You are getting confused between different things I think.

-

"Too much is almost enough. Anything in life worth doing is worth overdoing. Moderation is for cowards."
Mfusick is online now  
post #2441 of 3764 Old 01-03-2014, 06:43 PM
AVS Addicted Member
 
assassin's Avatar
 
Join Date: Jul 2004
Posts: 12,894
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 205
Quote:
Originally Posted by Mfusick View Post

If you make changes you should re-run update I agree with. But if it happens automatically in less then 24 hours you might just wait it out is my approach. I don't re-run updates after making changes. I see no point. But in the longer validate or verify processes I think you can't change data or it will throw an error or mess with the integrity. You are getting confused between different things I think.

No I am not. I have tested this all out. The worst case scenario is that FlexRaid will tell you it failed validate or verify in which case you run it again after an update.

Again, I am sure you have unknowingly done this if you use your server as you say you do. And again if this really was the case it would render this software as unusable as there is absolutely no way to know when/if you are in the middle of an update/verify/validate on the other end (the HTPC or remote desktop interface or your desktop or your wife's laptop, etc).
assassin is offline  
post #2442 of 3764 Old 01-03-2014, 06:52 PM - Thread Starter
AVS Addicted Member
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 21,724
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 79 Post(s)
Liked: 683
Quote:
Originally Posted by bryansj View Post

I had found that link before I posted. It explained what it did, but nothing really mentions how often one should run them. I had found other users post the 1d, 1w, 1m schedule, but had no official word. I'll consider your post the official word.

Does a verify and/or validate fail if data is added to the DRU during the checking process? As an example I have an IP cam that writes the video motion alerts to the pool. Should I bother moving the folder out of the pool?
Quote:
Originally Posted by spectrumbx View Post

Not unless the cam is appending to an existing file (vs a new file).
Changed data will affect Validate and Verify. New data coming in won't.
Quote:
Originally Posted by Mfusick View Post

So adding a file won't effect a verify or validate ???
Quote:
Originally Posted by spectrumbx View Post

No, it won't.

Quote:
Originally Posted by spectrumbx View Post

It really depends on what the array contains.
For instance, I have an array made up of movies, full backup images, and zipped up VMs. RAID-F's Snapshot RAID has been working fantastic for me on that array.
I have another array where I have downloads, software data, documents, and whatnot. On that array, I was using RAID-F's RT RAID until tRAID now.

In all, I would recommend Transparent RAID above everything else as it works equally well for real-time data as it does for snapshot data.
Most users will typically only want one array and not want to do the separation of snapshot data vs real-time data. Basically, most users will want to configure an all purpose massive array.
Snapshot RAID is great, but when you find yourself having to do a lot of daily updates, then a real-time solution might be best.
So, even if your data is 95% snapshot data and only 5% real-time data, you might not feel like separating them. In which case, tRAID works the best.

Snapshot RAID is more flexible no question asked; supporting network drives, flash disks, specific folders or files, file exclusions, partial and fuzzy recovery in worst cases, etc.
However, I think Transparent RAID has a much greater edge for being a "one size fits all". smile.gif

Basically just don't go changing data during VALIDATE or VERIFY and you should be all set. If you really wanted to run UPDATE you could after every change you make on your server but if it does it automatically every 24 hours I would not bother doing this.

This all goes back to my original point where if validate or verify take a long time then you basically should not be using your server during that time or you might disrupt the integrity. Thus, shorter parity calculation is more ideal. Does not mean anything else is NOT IDEAL. Just less ideal wink.gif

If you data is mostly stagnant (like media) and you can suffer the 24 hour limitation on updating then snapshot is a great choice. Or you can set update to run every 12 hours, or every 6 hours or whatever you wanted. But at a point you really should be using real time.

As long as my parity can finish before I might want to use my server then I think snapshot is a good choice. If you start filling up slow 4TB drives then snapshot takes a pretty long time to do a verify or validate process, which might not finish by morning if you set it to start at 1am.

-

"Too much is almost enough. Anything in life worth doing is worth overdoing. Moderation is for cowards."
Mfusick is online now  
post #2443 of 3764 Old 01-03-2014, 07:01 PM
AVS Addicted Member
 
assassin's Avatar
 
Join Date: Jul 2004
Posts: 12,894
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 205
Right. Incoming data will not affect the validate or verify but it also won't be included in the parity drive until you run "update" again. I never stop and think "Hey. I can't use my server. It might be running something". And I have never had a problem.

And I guess that's the whole point and another thing I love about FlexRaid.
assassin is offline  
post #2444 of 3764 Old 01-03-2014, 07:03 PM - Thread Starter
AVS Addicted Member
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 21,724
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 79 Post(s)
Liked: 683
Quote:
Originally Posted by assassin View Post

No I am not. I have tested this all out. The worst case scenario is that FlexRaid will tell you it failed validate or verify in which case you run it again after an update.

Again, I am sure you have unknowingly done this if you use your server as you say you do. And again if this really was the case it would render this software as unusable as there is absolutely no way to know when/if you are in the middle of an update/verify/validate on the other end (the HTPC or remote desktop interface or your desktop or your wife's laptop, etc).

I do know because I can see all the LED lights dancing on my NORCO. And- if I go messing around changing meta data or renaming movies or ripping a BR to the array it will throw an error.

And- To my first point and the one you disagree with -- I DO NOT WANT TO RE-RUN VALIDATE OR VERIFY. They take long enough to do, so I don't want to do it twice. And, I do not want to not be able to do stuff while they are running. I don't want to wait for them to finish to do something. I do not want to have to re-run them because I got an error.

This is exactly why I like making sure my server can complete all this stuff in off hours. Using slow 4TB hard drives, as opposed to faster cheaper 3TB 7200rpm drives only exacerbates the problem.

In an odd way you continue to support my opinions unintentionally.

Why would I want to have to re-run it again because I did not realize it was running when I did something the first time ? That's exactly what I have been telling you I want to avoid. You make it seem like no big deal, just re-run it. I think it's easier to just design your system so it's never going to be an issue. That is basically what I have been saying all along and you seem disagree with me about.

-

"Too much is almost enough. Anything in life worth doing is worth overdoing. Moderation is for cowards."
Mfusick is online now  
post #2445 of 3764 Old 01-03-2014, 07:07 PM
AVS Addicted Member
 
assassin's Avatar
 
Join Date: Jul 2004
Posts: 12,894
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 205
Quote:
Originally Posted by Mfusick View Post

I do know because I can see all the LED lights dancing on my NORCO. And- if I go messing around changing meta data or renaming movies or ripping a BR to the array it will throw an error.

And- To my first point and the one you disagree with -- I DO NOT WANT TO RE-RUN VALIDATE OR VERIFY. They take long enough to do, so I don't want to do it twice. And, I do not want to not be able to do stuff while they are running. I don't want to wait for them to finish to do something. I do not want to have to re-run them because I got an error.

This is exactly why I like making sure my server can complete all this stuff in off hours. Using slow 4TB hard drives, as opposed to faster cheaper 3TB 7200rpm drives only exacerbates the problem.

In an odd way you continue to support my opinions unintentionally.

You missed my point where I said "worst case scenario". I also said that this is not a problem and it doesn't have to be done. I can't remember the last time I have had to re-do a validate or verify and I have a wife and 3 kids that use my server almost all day long and have definitely used it during validate and verify calculations.

Also, most people aren't watching the lights on their Norco. Their server more than likely is tucked away in a remote location and they have no idea what it is doing 99% of the time if they are using it as a media playback device.
assassin is offline  
post #2446 of 3764 Old 01-03-2014, 07:16 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,047
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 124 Post(s)
Liked: 255
From the flexraid wiki...

http://wiki.flexraid.com/2011/10/18/understanding-the-limitations-of-snapshot-raid/
Quote:
The only operations that you should concern yourself with are those that might compromise the recovery process if a drive was to ever fail while the RAID is synchronized. FlexRAID knows how to deal with some of these operations and recover 100% despite them.
  • Renames will NEVER compromise recovery
  • Moves including moves to different drives that are part of the RAID will NEVER compromise recovery
  • Deletes will compromise recovery UNLESS the operations are done through the storage pool and FlexRAID’s proprietary recycle bin feature is turned on
  • Edits WILL compromise recovery
  • Writing new files to your RAID will never compromise recovery, but you risk losing these new files (and only those new files) if the drive they are on fails before you have a chance to synchronize the RAID

So you're at risk from losing data with Deletes, Adds and Edits.

Sounds to me like Mfusick falls under the usage pattern of someone that would find realtime raid ideal.

If I had a signature, this is where it would be.
ajhieb is online now  
post #2447 of 3764 Old 01-03-2014, 07:23 PM - Thread Starter
AVS Addicted Member
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 21,724
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 79 Post(s)
Liked: 683
Quote:
Originally Posted by ajhieb View Post

From the flexraid wiki...

http://wiki.flexraid.com/2011/10/18/understanding-the-limitations-of-snapshot-raid/
So you're at risk from losing data with Deletes, Adds and Edits.

Sounds to me like Mfusick falls under the usage pattern of someone that would find realtime raid ideal.

No I think you can do deletes as long as they are through the storage pool and you have recycle bin on just like it says. You can add (copy to) or "writes" as well, but if you added stuff since your last update you will lose it. But anything before your last update is fine. Moves and renames are ok too.

So like I was saying it's basically changes or "edits" that are the problem. I don't do much of that. I don't mind knowing anything I might have done in the last 24 hours might be lost in the case of a drive failure either. That is why snapshot works fine for me.
Quote:
Originally Posted by assassin View Post

You missed my point where I said "worst case scenario". I also said that this is not a problem and it doesn't have to be done. I can't remember the last time I have had to re-do a validate or verify and I have a wife and 3 kids that use my server almost all day long and have definitely used it during validate and verify calculations.

Also, most people aren't watching the lights on their Norco. Their server more than likely is tucked away in a remote location and they have no idea what it is doing 99% of the time if they are using it as a media playback device.

Ok you are just jumping all over the place in a wishy-washy stance.

First you say it's not a big deal about parity calculation process time and that it's not important that it is done finished before you use your server. Then you say you will get an error and need to re-run it. Then you say you can't remember the last time you got an error suggesting it won't happen.

Make up your mind.

Here's the facts:

If you go messing around on your server while it's running VALIDATE or VERIFY you probably are going to get an error (I have). By messing around I mean in the context of a media server, so ripping a BR disc to it, running MediaCenterMaster and updating or adding meta data, art, or renaming video files, or adding, deleting, moving around movies or tv shows. Basically whatever it is you might do with a media server. If VALIDATE or VERIFY are done and complete you can have it without any worries.

Thus, this is why it is IDEAL to have these functions finish in off periods of time when you won't be messing around with your server. This is also why if these processes are still running when you go and want to use your flexraid media server it's LESS IDEAL.
I basically have been saying this the entire time. If you can, design your system ideally and you never need to worry about this stuff.

-

"Too much is almost enough. Anything in life worth doing is worth overdoing. Moderation is for cowards."
Mfusick is online now  
post #2448 of 3764 Old 01-03-2014, 07:24 PM
AVS Addicted Member
 
assassin's Avatar
 
Join Date: Jul 2004
Posts: 12,894
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 205
Quote:
Originally Posted by Mfusick View Post

Ok you are just jumping all over the place in a wishy-washy stance.

First you say it's not a big deal about parity calculation process time and that it's not important that it is done finished before you use your server. Then you say you will get an error and need to re-run it. Then you say you can't remember the last time you go an error.
Which is it ?

Here's the facts:

If you go messing around on your server while it's running VALIDATE or VERIFY you probably are going to get an error (I have). By messing around I mean in the context of a media server, so ripping a BR disc to it, running MediaCenterMaster and updating or adding meta data, art, or renaming video files, or adding, deleting, moving around movies or tv shows. Basically whatever it is you might do with a media server. If VALIDATE or VERIFY are done and complete you can have it without any worries.

Thus, this is why it is IDEAL to have these functions finish in off periods of time when you won't be messing around with your server. This is also why if these processes are still running when you go and want to use your flexraid media server it's LESS IDEAL.
I basically have been saying this the entire time. If you can, design your system ideally and you never need to worry about this stuff.

No this is not at all what I said. I have altered my data many times during a validate or verify and not received an error.

What I said is that for some reason you do get an error you can run an update and then re-verify/validate. Like I said otherwise the software would be unusable as how would the hundreds/thousands of people using a flexraid server have any idea when/if their server was running one of these?

I am done arguing this. People can use whatever information they want. Its the internet after all and we all know that everything you read on the internet is true.
assassin is offline  
post #2449 of 3764 Old 01-03-2014, 07:33 PM
AVS Special Member
 
kapone's Avatar
 
Join Date: Jan 2003
Posts: 4,313
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 9 Post(s)
Liked: 39
Quote:
Originally Posted by assassin View Post

everything you read on the internet is true.
Couldn't agree more! And I played golf with that Obama dude last weekend!
kapone is offline  
post #2450 of 3764 Old 01-03-2014, 07:35 PM - Thread Starter
AVS Addicted Member
 
Mfusick's Avatar
 
Join Date: Aug 2002
Location: Western MA
Posts: 21,724
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 79 Post(s)
Liked: 683
Quote:
Originally Posted by assassin View Post

No this is not at all what I said. I have altered my data many times during a validate or verify and not received an error.

What I said is that for some reason you do get an error you can run an update and then re-verify/validate. Like I said otherwise the software would be unusable as how would the hundreds/thousands of people using a flexraid server have any idea when/if their server was running one of these?

I am done arguing this. People can use whatever information they want. Its the internet after all and we all know that everything you read on the internet is true.

Don't do that. Don't pretend like altering your data and messing around with your server during parity process like validate or verify is ok just because you want to seem right. You preach you want to provide the proper information and that is why you argue so much, then go and twist things around to support what you say ? Pick up your ball and go home if you want, but don't suggest misinformation.

Fact:
You should not mess around with your media server during parity computation. Do it when these are completed.

And... it will throw an error. Many people including me have gotten errors for these reasons, so please do not suggest it won't.

This entire argument started because I expressed cheaper faster smaller 7200rpm hard drives might be more ideal and larger slower hard drives might be less ideal. I said that because it increases parity compute times which could make your server busy when you wanted to use it, and that is best avoided. You cry about parity computation times and performance don't matter and then after all this arguing around in circles I find it funny it comes back full circle to everything I said from the beginning.

-

"Too much is almost enough. Anything in life worth doing is worth overdoing. Moderation is for cowards."
Mfusick is online now  
post #2451 of 3764 Old 01-03-2014, 08:43 PM
Senior Member
 
Gronnie's Avatar
 
Join Date: Aug 2010
Location: Rochester, MN
Posts: 249
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 16 Post(s)
Liked: 30
My FlexRaid server is exclusively 4TB 5900RPM drives and it has no problem doing update and verify and/or validate overnight.
Gronnie is offline  
post #2452 of 3764 Old 01-03-2014, 09:40 PM
Advanced Member
 
jim2100's Avatar
 
Join Date: Aug 2011
Posts: 762
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 33
Quote:
Originally Posted by ajhieb 
http://wiki.flexraid.com/2011/10/18/understanding-the-limitations-of-snapshot-raid/
So you're at risk from losing data with Deletes, Adds and Edits.

Well, the worst risk is when you modify (edit) the contents of an existing file. The reason is that such action compromises the ability to recover some data on other drives if another drive (not the one with the modified data) fails before you update the parity.

If you add a file, the only risk is to the added file. If you delete a file "properly", which means it gets moved to a temporary trash bin until parity is updated, then there is no risk with deleting a file.

Moving a file from one drive to another is essentially a delete and an add, so that has already been covered above.

The least risky operation is a move within a drive (or a rename), which is never a risk since the file can still be located by the filesystem pointer / inode.

By the way, with snapshot RAID and a media collection, I think it is prudent to exclude / filter all the metadata files from parity, since several of the media management programs frequently modify metadata, and as I explained above, modifying an existing file is the riskiest operation with snapshot RAID. Better just not to include the metadata in the parity computation, since the metadata is easily replaced (unless you have custom generated metadata, in which case you need another strategy).
jim2100 is offline  
post #2453 of 3764 Old 01-03-2014, 10:00 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,047
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 124 Post(s)
Liked: 255
Quote:
Originally Posted by jim2100 View Post

Well, the worst risk is when you modify (edit) the contents of an existing file. The reason is that such action compromises the ability to recover some data on other drives if another drive (not the one with the modified data) fails before you update the parity.

If you add a file, the only risk is to the added file. If you delete a file "properly", which means it gets moved to a temporary trash bin until parity is updated, then there is no risk with deleting a file.

Right. Adding a file has less risk, but there is still risk involved. What I'm curious about is the delete operation. Is it possible to "properly" delete files from a flexraid volume via the network (mapped drive, UNC path... whatever) or must you do it while logged into the server?

Quote:
Moving a file from one drive to another is essentially a delete and an add, so that has already been covered above.

Same question as above... does that still apply if done remotely or do I need to move the file from within the flexraid framework?

If I had a signature, this is where it would be.
ajhieb is online now  
post #2454 of 3764 Old 01-03-2014, 10:00 PM
AVS Addicted Member
 
assassin's Avatar
 
Join Date: Jul 2004
Posts: 12,894
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 205
Quote:
Originally Posted by jim2100 View Post

Well, the worst risk is when you modify (edit) the contents of an existing file. The reason is that such action compromises the ability to recover some data on other drives if another drive (not the one with the modified data) fails before you update the parity.

If you add a file, the only risk is to the added file. If you delete a file "properly", which means it gets moved to a temporary trash bin until parity is updated, then there is no risk with deleting a file.

Moving a file from one drive to another is essentially a delete and an add, so that has already been covered above.

The least risky operation is a move within a drive (or a rename), which is never a risk since the file can still be located by the filesystem pointer / inode.

By the way, with snapshot RAID and a media collection, I think it is prudent to exclude / filter all the metadata files from parity, since several of the media management programs frequently modify metadata, and as I explained above, modifying an existing file is the riskiest operation with snapshot RAID. Better just not to include the metadata in the parity computation, since the metadata is easily replaced (unless you have custom generated metadata, in which case you need another strategy).

That's my understanding as well. And this is only an issue if you have a drive fail. But its completely different than "throwing an error" which I don't quite understand. And, again, if you are updating your drives every day this risk is pretty minimal, imo, compared to many other options. That's also why I continue to be be careful about using the term "redundancy" and not "backup".

Also per the wiki files that are changed "infrequently" are not considered "edits".
Quote:
Use of Snapshot RAID
- Data that is edited infrequently (move, rename, add, and delete [if recycle bin mode is on] are not edits and are safe operations)
assassin is offline  
post #2455 of 3764 Old 01-03-2014, 10:04 PM
Senior Member
 
rc05's Avatar
 
Join Date: Oct 2011
Location: San Luis Obispo, CA
Posts: 327
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 17 Post(s)
Liked: 46
Quote:
Originally Posted by Mfusick View Post

Fact:
You should not mess around with your media server during parity computation. Do it when these are completed.

And... it will throw an error. Many people including me have gotten errors for these reasons, so please do not suggest it won't.

I've got a TV show that records at the same time as my Update task at 4 AM. It gets dumped onto the array during the update operation and no error has ever been thrown. But I *think* (I'm not 100% cause I haven't checked) that it isn't included in the concurrent Update.
rc05 is offline  
post #2456 of 3764 Old 01-03-2014, 10:06 PM
Senior Member
 
rc05's Avatar
 
Join Date: Oct 2011
Location: San Luis Obispo, CA
Posts: 327
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 17 Post(s)
Liked: 46
Quote:
Originally Posted by ajhieb View Post

What I'm curious about is the delete operation. Is it possible to "properly" delete files from a flexraid volume via the network (mapped drive, UNC path... whatever) or must you do it while logged into the server?

If by "proper" you mean the "recycle bin" feature is turned on, deleting files from a network share will correctly move the file to flexraid's "recycle bin".
rc05 is offline  
post #2457 of 3764 Old 01-03-2014, 10:10 PM
AVS Addicted Member
 
assassin's Avatar
 
Join Date: Jul 2004
Posts: 12,894
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 205
Quote:
Originally Posted by Gronnie View Post

My FlexRaid server is exclusively 4TB 5900RPM drives and it has no problem doing update and verify and/or validate overnight.

Thanks for sharing your firsthand experience. I haven't made jump to 4TB yet but am considering it. Nice to know it isn't much different than my current setup.
assassin is offline  
post #2458 of 3764 Old 01-03-2014, 10:12 PM
AVS Addicted Member
 
assassin's Avatar
 
Join Date: Jul 2004
Posts: 12,894
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 205
Quote:
Originally Posted by rc05 View Post

I've got a TV show that records at the same time as my Update task at 4 AM. It gets dumped onto the array during the update operation and no error has ever been thrown. But I *think* (I'm not 100% cause I haven't checked) that it isn't included in the concurrent Update.

It won't be included in that night's update. But it will included in the next night's update.
assassin is offline  
post #2459 of 3764 Old 01-03-2014, 10:21 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,047
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 124 Post(s)
Liked: 255
Quote:
Originally Posted by rc05 View Post

If by "proper" you mean the "recycle bin" feature is turned on, deleting files from a network share will correctly move the file to flexraid's "recycle bin".

You mean it's optional?!? Is there a reason you'd want that turned off?

To get into the technical side of things, (and since the flexraid wiki isn't up at the moment... must be doing a validate) is that the reason that you also have to create network shares through flexraid as well as Windows? is it intercepting calls to the file system API and rerouting things to make sure the parity still jives when a file is deleted?

Would that also imply that the space from the deleted file(s) isn't freed until an update is performed?

If I had a signature, this is where it would be.
ajhieb is online now  
post #2460 of 3764 Old 01-03-2014, 10:25 PM
Senior Member
 
rc05's Avatar
 
Join Date: Oct 2011
Location: San Luis Obispo, CA
Posts: 327
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 17 Post(s)
Liked: 46
Quote:
Originally Posted by ajhieb View Post

You mean it's optional?!? Is there a reason you'd want that turned off?

To get into the technical side of things, (and since the flexraid wiki isn't up at the moment... must be doing a validate) is that the reason that you also have to create network shares through flexraid as well as Windows? is it intercepting calls to the file system API and rerouting things to make sure the parity still jives when a file is deleted?

Would that also imply that the space from the deleted file(s) isn't freed until an update is performed?

Yeah, for some reason, the default is the "recycle bin" feature turned off. You are correct, the space from the deleted files isn't freed until Update is run. That could be one reason why you wouldn't want it on, but I personally don't care about that and l have "recycle bin" turned on. What's happening is that it moves the deleted files into a hidden folder and excludes that folder from the Update operation. After the Update operation is successful, it actually deletes the folder.

If you go into Flexraid Drive Manager and see the drive contents, you can see the deleted files in the hidden _flxr_/d/ folder. If you accidentally delete something, you can shut down the pool, mount the drive, and retrieve it even (if Update hasn't been run yet of course).

I'm not sure the reason why you have to set the network shares in Flexraid though.
Mfusick likes this.
rc05 is offline  
Reply Home Theater Computers

Tags
Flexraid , Intel Core I5 4670k 3 4ghz Lga 1150 Quad Core Desktop Processor , Intel Core I3 4130 3 4 3 Fclga 1150 Processor Bx80646i34130 , Asrock , Asrock Z87m Extreme4 Lga1150 Intel Z87 Chipset Ddr3 Quad Crossfirex Quad Sli Sata3 Usb3 0 Microatx M , Asus , Asus Us , Asus Computer , Asus Components , Norco , Asus Computer International Direct , Flexraid Raid F , Seagate , Seagate Hard Drives , Seagate Freeagent Theater 1080p Hd Media Player Stcea201 Rk , Hitachi , Dell , Windows 7 Vista Xp Media Center Mce Pc Remote Control And Infrared Receiver , Hp Oem Window Media Center Mce Pc Remote Control And Infrared Receiver For Windows7 Vista Xp Home Pr , Intel , Intel Core I5 3570 3 4 Ghz Processor , Intel Pentium G2020 2 9ghz Lga 1155 Dual Core Desktop Processor , Intel Pentium G3220 3 0ghz Lga 1150 Dual Core Desktop Processor , Amd , Amd A10 6700 Richland 4 2ghz Socket Fm2 65w Quad Core Desktop Processor Amd Radeon Hd Ad6700okhlbox , Amd A6 5400k 3 6ghz Socket Fm2 Dual Core Desktop Processor , Amd A8 5600k 3 6ghz Socket Fm2 Quad Core Desk
Gear in this thread

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off