Drive Pool solution or something else? Alternatives? - AVS Forum
Forum Jump: 
Reply
 
Thread Tools
post #1 of 49 Old 10-24-2013, 09:42 AM - Thread Starter
Newbie
 
JazzJackRabbit's Avatar
 
Join Date: Oct 2013
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 2 Post(s)
Liked: 10

In case you're wondering this is yet another drive pool thread, but this one has a little twist. I think? So please keep reading.

 

I'll try to keep this as brief as I can. About 6 years ago I've built a 7TB (8TB total, 1TB lost to parity) hardware RAID5 array on Adaptec 31605 card. I have since outgrew the array capacity and simply started putting my files on a bunch of drives on yet another file server. As you can imagine it was kind of a nightmare, not only half of my data was essentially JBOD with no redundancy whatsoever, but my data was also spread across 2 different computers across multiple hard drives.

In an effort to reorganize this whole mess and make sure it's all redundant I'm currently in the process of upgrading my fileserver. I have decided that another hardware RAID array would not be a good solution for me, so I opted for a JBOD configuration with SnapRAID for parity protection. I have purchased 5 4TB seagates, copied all of my data to the new drives, and dedicated 1 drive to parity. I have downloaded SnapRAID and it's currently calculating parity. I still want to get a second drive for dual parity, and one or two more drives for data, but as of right now the new system is live. Wooooooo!

 

Just about the only thing left to do was to find a good drive pooling solution. I have been reading a lot in the past week about drivepool and drivebender, liquesce, etc. I have finally settled on DrivePool, so I installed it yesterday and gave it a whirl. And this is where I ran into a problem... Apparently I totally misunderstood how DrivePool would work. What I wanted to do is this. Because my data is mostly static I wanted to make the drive pool "read only" and manually manage my data on my hard drives. I wanted to manually copy the data to the data drives on my file server and I wanted the drive pool to simply "pool" all the data from my hard drives and represent it as one big hard drive I could access anywhere from the network inside my place. I did not want the drive pool to manage my data, I did not want it to duplicate data, I did not want it to rebalance it. I wanted to balance the data across the hard drive manually myself, and I wanted to have SnapRAID provide redundancy. The automatic drive pool rebalancing would just increase the chances of something failing because SnapRAID does not calculate parity in real time. Basically I wanted to copy data to the individual hard drives and have the drive pool pick up the changes, pool all of the drives together, and immediately show newly copied files in one big pool. And this is not how DrivePool works. As I have learned, I cannot do that, I cannot just copy data to the individual hard drives and have that data show up in the pool. In order for any data to be available in the Pool, I actually have to copy data into the Pool and have the pool manage the data across hard drives. This is not what I wanted originally because it means it would be up to the pool software to manage my data across the hard drives and rebalance it if necessary.

 

I've been looking more closely into DriveBender and it appears it works in the same manner, i.e. you have to copy files onto the pool instead of individual hard drives, but I cannot tell for sure without giving it a try. No idea on Liquesce, that one has even less documentation than the other two.

 

So can anyone suggest a solution to my problem? I want to:
- Manually manage data on the individual hard drives
- I want my pool software to simply pool all the individual hard drives into one big read only pool with NO balancing/redundancy management whatsoever
- I would prefer the changes to the data on individual hard drives reflected immediately in the pool

 

Is there anything that would work the way I want to? Or am I stuck with using junctions/symlinks/hardlinks? BTW I'm still not sure on the differences between the three, so if anyone could chime on using those, and which one would be the best solution for me, I would appreciate that too.

 

 

==========================

 

 

It seems that the thread conversation is starting to veer off in all the different directions that are tangent to what I want to do, so I'd like to summarize what I need in a bunch of small bullet points. I want:
- Fault tolerance with two parity drives and I do not want my data striped, essentially I want software RAID4 with two parity drives which can be achieved by either RAID-F(formelly FlexRAID) or SnapRAID
- I must retain access to the individual hard drives so that I can write data to them
- I must be the only one to have write privileges to the raw hard drives
- Rest of the people in the house need to be given write access to specific folders only to limit the damage they can do
- I must be able to access all of the data on my hard drives from a single place as if all of this data was "merged" into a single folder, i.e. if I have four drives

 

Drive 1
Backup
Movies

 

Drive 2
Anime
Movies

 

Drive 3
Music
Movies

 

Drive 4
Series
Movies

 

I need to be able to access all of the folders from one place, that is that place should look like this:

Anime
Backup
Movies
Music
Series

 

- Last bullet point, that place/pool/whatever should be read only to minimize accidental damage

JazzJackRabbit is offline  
Sponsored Links
Advertisement
 
post #2 of 49 Old 10-24-2013, 10:11 AM
AVS Special Member
 
StardogChampion's Avatar
 
Join Date: Dec 2007
Location: New Hampshire, USA
Posts: 2,995
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 31 Post(s)
Liked: 117
I was under the impression you could turn off the balancing and redundancy. You can also add existing drives to the pool. So, if you had the drives setup the way you wanted and then add them to the pool, adding new files to a folder in the pool should just add them where the folder originally resides.

I am planning to use DrivePool with just pooling/rebalancing and do nightly backups of certain pool folders to a drive outside the pool instead of relying on redundancy since it doesn't protect against accidental deletion, the most likely culprit in my house of losing data.

FlexRAID would allow you to pool the drives, keep the folders where they reside and provide parity.

 

 

StardogChampion is offline  
post #3 of 49 Old 10-24-2013, 10:24 AM - Thread Starter
Newbie
 
JazzJackRabbit's Avatar
 
Join Date: Oct 2013
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 2 Post(s)
Liked: 10

Yes, with DrivePool at least, I can turn off balancing and I can turn off redundancy. And I can also add existing drives to the pool with data on them. However, any existing files on the hard drives will not appear in the pool, and any files I copy to individual hard drives will also not appear in the pool. Bottom line is, if I want to use DrivePool, I have to copy data in the "pool".

 

Here's what the DrivePool manual says on adding files to the individual hard drives vs the pool:
http://stablebit.com/Support/DrivePool/2.X/Manual?Section=Adding%20a%20Drive%20to%20the%20Pool%20/%20Creating%20a%20New%20Pool

 

More specifically this quote:
"When you copy files to any drive, even when it's part of the pool, the files are stored on that drive and not on the pool."

 

Basically, it means that if I want to use the DrivePool I have to copy data into the pool, and not on individual drives. Depending on where DrivePool keeps the data I may also not be able to use SnapRAID with it. I haven't tested it, but DrivePool creates hidden folder on each drive in the pool, and it is my suspicion that anything I copy into the pool gets placed into those hidden folders. The problem is SnapRAID by default ignores any hidden files and folders and there is no way to override this behavior, so my data will be unprotected. Sigh......

JazzJackRabbit is offline  
post #4 of 49 Old 10-24-2013, 10:36 AM
Member
 
Mark Guebert's Avatar
 
Join Date: Dec 2001
Location: St. Louis, Missouri
Posts: 145
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by JazzJackRabbit View Post

Yes, with DrivePool at least, I can turn off balancing and I can turn off redundancy. And I can also add existing drives to the pool with data on them. However, any existing files on the hard drives will not appear in the pool, and any files I copy to individual hard drives will also not appear in the pool. Bottom line is, if I want to use DrivePool, I have to copy data in the "pool".

Here's what the DrivePool manual says on adding files to the individual hard drives vs the pool:
http://stablebit.com/Support/DrivePool/2.X/Manual?Section=Adding%20a%20Drive%20to%20the%20Pool%20/%20Creating%20a%20New%20Pool

More specifically this quote:

"When you copy files to any drive, even when it's part of the pool, the files are stored on that drive and not on the pool."

Basically, it means that if I want to use the DrivePool I have to copy data into the pool, and not on individual drives. Depending on where DrivePool keeps the data I may also not be able to use SnapRAID with it. I haven't tested it, but DrivePool creates hidden folder on each drive in the pool, and it is my suspicion that anything I copy into the pool gets placed into those hidden folders. The problem is SnapRAID by default ignores any hidden files and folders and there is no way to override this behavior, so my data will be unprotected. Sigh......


Sounds to me like Flexraid would be a more applicable solution as it is a all in one solution to your needs. Flexraid allows you to add individual drives with data already on them and will not touch the data. Futher, once the pool is formed you can dictate how further information is added to the pool based on a folder or drive space priority. It also provides parity protection as well. I did exactly as you. When I formed my pool I had around 10tb of data, I kept my existing data and folder structure, formed the pool and created parity all in one shot. Now I don't care how the data is organized because I have the parity protection and I have set folder priority.
Mark Guebert is offline  
post #5 of 49 Old 10-24-2013, 02:11 PM
Advanced Member
 
jim2100's Avatar
 
Join Date: Aug 2011
Posts: 762
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 33
With linux, it is almost trivially easy to create such a read-only drive pool with symlinks (symbolic links). The only thing it would not do would be to automatically add symlinks for new files (but you could create a cron job to do that, say every hour, if you wanted...or just a script that you run manually whenever you add new files).

Now, Windows also has symlinks (and junctions), but based on some discussions I have seen, there are difficulties with using symlinks to make a read-only pool in Windows. I don't fully understand the issues, but I have seen people posting that a Windows symlink pool does not work properly over a network share. This was discussed in the SnapRAID forum some time ago. I've been a bit skeptical whether it is really a Windows limitation that you cannot make such a symlink pool that will work over a network share (or whether it is just a misunderstanding). Hopefully someone more familiar than me with how Windows does symlinks and junctions can explain the issues.

http://msdn.microsoft.com/en-us/library/aa363878%28v=vs.85%29.aspx

http://msdn.microsoft.com/en-us/library/aa365006%28v=vs.85%29.aspx
jim2100 is offline  
post #6 of 49 Old 10-24-2013, 06:26 PM
AVS Special Member
 
StardogChampion's Avatar
 
Join Date: Dec 2007
Location: New Hampshire, USA
Posts: 2,995
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 31 Post(s)
Liked: 117
The symlink strategy is also what Greyhole uses: http://www.greyhole.net/

 

 

StardogChampion is offline  
post #7 of 49 Old 10-24-2013, 09:22 PM
AVS Special Member
 
lespurgeon's Avatar
 
Join Date: May 2003
Location: California
Posts: 2,455
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 40
I'm confused what exactly you want to do and why. The idea of a drive pool isn't to individually manage what file is on which disk.
Flexraid may be the best solution - and use it for your parity needs as well
I've also considered drive bender and drive pool to create a storage group where if one disk is lost only the data on that disk is lost (I can always re-rip movie disks that are stored away) since I wouldn't want to lose 4 drives worth of data, as would happen with Windows built-in solution.
lespurgeon is offline  
post #8 of 49 Old 10-24-2013, 10:13 PM
Advanced Member
 
jim2100's Avatar
 
Join Date: Aug 2011
Posts: 762
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 33
Quote:
Originally Posted by lespurgeon View Post

I'm confused what exactly you want to do and why. The idea of a drive pool isn't to individually manage what file is on which disk.

I'm not sure why you are confused. He described what he wants quite clearly. A read-only pool, with the ability for him to manually write files to drives when needed.

I use just such a method to manage my files on my linux server. A read-only pool, with files that I add to individual drives directly when needed, is quite convenient for a media fileserver.
jim2100 is offline  
post #9 of 49 Old 10-24-2013, 10:16 PM
Advanced Member
 
jim2100's Avatar
 
Join Date: Aug 2011
Posts: 762
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 33
By the way, have you tried the SnapRAID pool function yet? It creates symlinks, and works quite well on linux. But there is some question of how well it works on Windows network shares. To the best of my knowledge, if you use one Windows machine to access the pool of symlinks shared over the network from another Windows machine, it should work. But if you use certain media appliances that are supposed to be able to access Windows shares, it might not work (apparently a lot of media appliances do not know how to properly access symlinks over a Windows network share).

I do not have any first hand experience with a symlink pool in Windows, so I would be interested to hear what you found if you tested it.
jim2100 is offline  
post #10 of 49 Old 10-24-2013, 11:51 PM
Member
 
Bumtrinket's Avatar
 
Join Date: Mar 2007
Posts: 18
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 11
Quote:
Originally Posted by jim2100 View Post

I'm not sure why you are confused. He described what he wants quite clearly. A read-only pool, with the ability for him to manually write files to drives when needed.

I use just such a method to manage my files on my linux server. A read-only pool, with files that I add to individual drives directly when needed, is quite convenient for a media fileserver.

I'm pretty sure this could easily be achieved using mhddfs - set up each individual drive as a Samba share with read/write permissions, and have the combined (pool) mount point as an additional share with read-only permissions.

You could then just whack SnapRaid on top if desired, with an additional parity drive (presumably this wouldn't be shared or part of the mhddfs pool).

Is this the way you're doing it?
Bumtrinket is offline  
post #11 of 49 Old 10-25-2013, 12:05 AM
Advanced Member
 
tman247's Avatar
 
Join Date: Sep 2007
Location: UK
Posts: 852
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 4 Post(s)
Liked: 27
Interesting. I'm happily using DrivePool (on WHS2011), which works well for me. I'm now curious about how SnapRAID could help protect the pool. It seems that SnapRAID protects individual disks (up to two failures), so If snapraid only sees 'one' large hard disk (the pool), can it actually protect this ok? If I had one disk failure in the pool my gut feeling is that snapRAID might not help with the protection. I could be wrong though.
tman247 is online now  
post #12 of 49 Old 10-25-2013, 12:14 AM
Advanced Member
 
jim2100's Avatar
 
Join Date: Aug 2011
Posts: 762
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 33
Quote:
Originally Posted by Bumtrinket View Post

Is this the way you're doing it?

No, as I said, I am just using symlinks. I prefer that to mhddfs (or aufs3)

By the way, I assume the OP is running his fileserver on Windows, based on the drive pooling software he mentioned, so mhddfs is not an option for him.
jim2100 is offline  
post #13 of 49 Old 10-25-2013, 08:31 AM - Thread Starter
Newbie
 
JazzJackRabbit's Avatar
 
Join Date: Oct 2013
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 2 Post(s)
Liked: 10
Quote:
Originally Posted by tman247 View Post

Interesting. I'm happily using DrivePool (on WHS2011), which works well for me. I'm now curious about how SnapRAID could help protect the pool. It seems that SnapRAID protects individual disks (up to two failures), so If snapraid only sees 'one' large hard disk (the pool), can it actually protect this ok? If I had one disk failure in the pool my gut feeling is that snapRAID might not help with the protection. I could be wrong though.

To answer your question, yes and no. Even if you have all of your drives pooled with DrivePool into one huge drives, your operating system still has access to individual drives in the system and so does SnapRAID. So SnapRAID can still protect your drives as long as you buy another hard drive that will be your parity drive and that will not be included in the pool. However, and this is where the "no" part comes in. I've played with DrivePool some more yesterday evening, and what DrivePool does it creates a hidden folder called PoolPart.GUID on each hard drive in the pool and any data you write to the pool gets stored in one of those hidden folders. SnapRAID by default ignores hidden files and folders, so anything you write to the pool will not be protected against hard drive failure. I see two potential ways to solve this problem. One, is to somehow make sure SnapRAID does not ignore hidden folders, but I have not found a way to do it yet. I'm not saying it's impossible, but so far I haven't found a way to do it. Or two, manually remove PoolPart.GUID hidden attribute which will make it visible to the SnapRAID. However, I have no idea how good of a solution this would be because for all I know DrivePool may reapply hidden attribute to the folder upon next restart and then SnapRAID will ignore the pool data again. I'm going to play around with this more when I have time.

JazzJackRabbit is offline  
post #14 of 49 Old 10-25-2013, 09:07 AM - Thread Starter
Newbie
 
JazzJackRabbit's Avatar
 
Join Date: Oct 2013
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 2 Post(s)
Liked: 10
Quote:
Originally Posted by lespurgeon View Post

I'm confused what exactly you want to do and why. The idea of a drive pool isn't to individually manage what file is on which disk.
Flexraid may be the best solution - and use it for your parity needs as well
I've also considered drive bender and drive pool to create a storage group where if one disk is lost only the data on that disk is lost (I can always re-rip movie disks that are stored away) since I wouldn't want to lose 4 drives worth of data, as would happen with Windows built-in solution.

Simply put, I won't be the only one using this pool. I live with my girlfriend, and we have parents/friends over every once in a while, and we might get a roommate as well. The pool has to be accessible to everyone, however I have seen how careless other people are when it comes to data, so I don't want to be pulling my hair out every time someone flicks a mouse and asks with a bemused expression on their face "hey, where did it go?" So my solution to this problem is I want to be the only one with the access to the raw drives and the only one with the write priviledges to them. On the other hand I also want the convenience of having all of my data across multiple hard drives show up in a single place. Right now with the way it is set up if I want to play a movie that starts 0 to D I have to go to one hard drive, if I want to play a movie starting from E to M I have to go to another hard drive, and if I want to play a movie that starts from N to Z I have to go to yet another hard drive. It's annoying and there has to be a better way, which is why I want it all to show up in a single place. I might have given the wrong impression in my OP, it is quite possible that what I need is not a pool, but something different, but since Pool is what people are familiar with, that's the term I used.

 

PS I have read up on FlexRAID and according to this guide http://linustechtips.com/main/topic/33510-how-to-install-and-setup-flexraid-on-your-windows-storage-system/ FlexRAID removes pooled hard drives from the windows explorer. I haven't actually played with FlexRAID yet, but I'm sure it simply removes drive letter assignments, so I could probably get access to raw hard drives back if I wanted. However, it seems like kind of a hack. And to be completely honest, I may have no choice at using FlexRAID, but reading up on FlexRAID it seems like it's one huge hack, managing through web interface, really? IMO that's asking for trouble, having to manage network shares through the interface again instead of built in Windows Share Management because pool service might be slow to start up and shares might not work otherwise? I just have a bad feeling about the whole thing, which is why I'm so reluctant to try it out... Plus the sole developer responsible for it is busy with tRAID and NZFS, the last update to FlexRAID (now called RAID-F) has been about 6 months ago.

JazzJackRabbit is offline  
post #15 of 49 Old 10-25-2013, 09:13 AM - Thread Starter
Newbie
 
JazzJackRabbit's Avatar
 
Join Date: Oct 2013
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 2 Post(s)
Liked: 10
Quote:
Originally Posted by jim2100 View Post

With linux, it is almost trivially easy to create such a read-only drive pool with symlinks (symbolic links). The only thing it would not do would be to automatically add symlinks for new files (but you could create a cron job to do that, say every hour, if you wanted...or just a script that you run manually whenever you add new files).

Now, Windows also has symlinks (and junctions), but based on some discussions I have seen, there are difficulties with using symlinks to make a read-only pool in Windows. I don't fully understand the issues, but I have seen people posting that a Windows symlink pool does not work properly over a network share. This was discussed in the SnapRAID forum some time ago. I've been a bit skeptical whether it is really a Windows limitation that you cannot make such a symlink pool that will work over a network share (or whether it is just a misunderstanding). Hopefully someone more familiar than me with how Windows does symlinks and junctions can explain the issues.

http://msdn.microsoft.com/en-us/library/aa363878%28v=vs.85%29.aspx

http://msdn.microsoft.com/en-us/library/aa365006%28v=vs.85%29.aspx

I've tried the Pool option in the SnapRAID. It works, and it creates symlinks in the specified folder that I can access if I'm remoted into the file server. However I'm having problems accessing them over the network. I have solved the "Symbolic link cannot be followed because it's type is disabled by using fsutil on the client http://stackoverflow.com/questions/229643/how-do-i-overcome-the-the-symbolic-link-cannot-be-followed-because-its-type-is/ however I still cannot access them over the network. I haven't tried running fsutil on the fileserver, maybe that's the problem. I'll be trying it this evening to see if it works.

JazzJackRabbit is offline  
post #16 of 49 Old 10-25-2013, 09:14 AM
AVS Special Member
 
StardogChampion's Avatar
 
Join Date: Dec 2007
Location: New Hampshire, USA
Posts: 2,995
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 31 Post(s)
Liked: 117
If all the files are on a server and your family/friends/etc. are accessing them via clients, you could make the shares they mount on those clients read-only. Shares can be "pooled" on the clients using libraries if they're Windows 7/8 clients. Only the server has the physical layout and you can grant yourself management of that.

 

 

StardogChampion is offline  
post #17 of 49 Old 10-25-2013, 09:48 AM
AVS Special Member
 
EricN's Avatar
 
Join Date: May 2002
Posts: 1,166
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 12 Post(s)
Liked: 154
How married to Windows are you? It sounds like FreeNAS or DSM may be a better fit for your needs. Everything you mention: old hardware, new drives, multi-user support, access permissions, snapshots, etc...all of that points to installing a full stack NAS OS on the old server and being done instead of screwing with a bunch of hacks. You are in the "too big for Shares, too small for DFS" gap that Microsoft doesn't cover well.
EricN is offline  
post #18 of 49 Old 10-25-2013, 09:58 AM
Member
 
tcs2tx's Avatar
 
Join Date: Feb 2005
Location: Chicago, IL
Posts: 188
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 11
Quote:
Originally Posted by JazzJackRabbit View Post

Simply put, I won't be the only one using this pool. ... So my solution to this problem is I want to be the only one with the access to the raw drives and the only one with the write priviledges to them. On the other hand I also want the convenience of having all of my data across multiple hard drives show up in a single place.

The quote above appears to be your underlying goal. If that is the case, I don't see it as a "pooling" issue. Instead, I view it as a "permissions" issue. I have a similar problem - wife, 4 kids, and 2 inlaws that want to be able to access content while I want to make sure that nothing on my server gets messed up. I address the problem with permissions, which are available as a form of management in all file systems. When it comes to accessing the data itself, you can "pool" information from multiple drives in a number of different ways. From the server(s) you can export shares (samba/CIFS, NFS, etc.). If you're only talking about being able to select and play media, you can easily achieve with MyMovies, MediaBrowser, XBMC, Plex, etc. as a front-end since they can each access/manage content from multiple sources.

To illustrate, I'll give an example close to my real system:

HTPC clients (all Win7) have a user "family" with the password "family' and each machine is set up to automatically login under the "family" user. The HTPC each have MCE autostarting with MyMovies installed pointing to content from multiple shares (listed below)

Desktop clients (all Macs) have four users:

1) dad (with a unique password)
2) mom (with a unique password)
3) kids (no password)
4) guest

Server has the following groups and members

parents - dad and mom
house - dad, mom, kids and media

Server has the following shares with group member permissions:

documents - dad and mom (read/write) and kids (read/write only for one subdirectory) and everyone else (no access, ex)
photos - dad and mom (read/write) and everyone else (read only)
tv - dad and mom (read/write) and everyone else (read only)
movies - dad and mom (read/write) and everyone else (read only)
tcs2tx is offline  
post #19 of 49 Old 10-25-2013, 11:04 AM - Thread Starter
Newbie
 
JazzJackRabbit's Avatar
 
Join Date: Oct 2013
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 2 Post(s)
Liked: 10
Quote:
Originally Posted by tcs2tx View Post


The quote above appears to be your underlying goal. If that is the case, I don't see it as a "pooling" issue. Instead, I view it as a "permissions" issue.

Yes, I can manage the permissions with multiple logins. It's a bit of a pain without creating domain but it's doable. But I still want to be able to see all of my files in a single share in explorer. For example I have 4 drives with the following folders:

 

Drive 1

Backup

Movies

 

Drive 2

Anime

Movies

 

Drive 3

Music

Movies

 

Drive 4

Series

Movies

 

I would still like to create pool/share/place/whatever that will allow me to access all of this content from one place so that I will see "merge" of all of the above, i.e.:

Anime

Backup

Movies

Music

Series

 

And I would like this pool/share/place/whatever to be readonly. If necessary I can give write permissions to specific places to specific people to restrict possible damage they may do, but overall, I would like the whole thing to be read only access.

JazzJackRabbit is offline  
post #20 of 49 Old 10-25-2013, 11:06 AM
Senior Member
 
Elpee's Avatar
 
Join Date: Dec 2007
Posts: 379
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 17
Quote:
Originally Posted by JazzJackRabbit View Post

I've tried the Pool option in the SnapRAID. It works, and it creates symlinks in the specified folder that I can access if I'm remoted into the file server. However I'm having problems accessing them over the network. I have solved the "Symbolic link cannot be followed because it's type is disabled by using fsutil on the client http://stackoverflow.com/questions/229643/how-do-i-overcome-the-the-symbolic-link-cannot-be-followed-because-its-type-is/ however I still cannot access them over the network. I haven't tried running fsutil on the fileserver, maybe that's the problem. I'll be trying it this evening to see if it works.
Waiting for your result.
Elpee is offline  
post #21 of 49 Old 10-25-2013, 11:32 AM
AVS Special Member
 
EricN's Avatar
 
Join Date: May 2002
Posts: 1,166
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 12 Post(s)
Liked: 154
Quote:
Originally Posted by JazzJackRabbit View Post

Yes, I can manage the permissions with multiple logins. It's a bit of a pain without creating domain but it's doable. But I still want to be able to see all of my files in a single share in explorer. For example I have 4 drives with the following folders: [...]

And I would like this pool/share/place/whatever to be readonly. If necessary I can give write permissions to specific places to specific people to restrict possible damage they may do, but overall, I would like the whole thing to be read only access.

Create a Spanned Volume. Then layer the file permissions and share permissions on top of it.
EricN is offline  
post #22 of 49 Old 10-25-2013, 11:53 AM - Thread Starter
Newbie
 
JazzJackRabbit's Avatar
 
Join Date: Oct 2013
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 2 Post(s)
Liked: 10
Quote:
Originally Posted by EricN View Post


Create a Spanned Volume. Then layer the file permissions and share permissions on top of it.


Spanned volumes are not fault tolerant, if one drive goes, everything goes, and I highly doubt SnapRAID would be able to protect me against spanned volume failure (I may be wrong).

JazzJackRabbit is offline  
post #23 of 49 Old 10-25-2013, 12:35 PM
AVS Special Member
 
Mrkazador's Avatar
 
Join Date: Jun 2005
Posts: 3,827
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 35 Post(s)
Liked: 248

I use snapraid and junction folders for my "drive pool". I can set specific users read only or read/write access to the shared junction folders. Link Shell Extension is the software I use to create the Junction folders, just a couple clicks and you're done.

Mrkazador is offline  
post #24 of 49 Old 10-25-2013, 12:42 PM
AVS Special Member
 
EricN's Avatar
 
Join Date: May 2002
Posts: 1,166
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 12 Post(s)
Liked: 154
A:
Quote:
Originally Posted by JazzJackRabbit View Post

Spanned volumes are not fault tolerant, if one drive goes, everything goes, and I highly doubt SnapRAID would be able to protect me against spanned volume failure (I may be wrong).

B:
Quote:
Originally Posted by JazzJackRabbit View Post

The pool has to be accessible to everyone, however I have seen how careless other people are when it comes to data, so I don't want to be pulling my hair out every time someone flicks a mouse and asks with a bemused expression on their face "hey, where did it go?"

The only way to tolerate case B is to have a backup. Once you have that, you only lose uptime in case A; you don't lose data.

You can save money up front by trying to prevent B and relying on RAID to save your data in case A, but it'll always cost more in the long run.
EricN is offline  
post #25 of 49 Old 10-25-2013, 02:08 PM
Member
 
tcs2tx's Avatar
 
Join Date: Feb 2005
Location: Chicago, IL
Posts: 188
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 11
Quote:
Originally Posted by JazzJackRabbit View Post

I would still like to create pool/share/place/whatever that will allow me to access all of this content from one place so that I will see "merge" of all of the above, i.e.:
Anime
Backup
Movies
Music
Series

And I would like this pool/share/place/whatever to be readonly. If necessary I can give write permissions to specific places to specific people to restrict possible damage they may do, but overall, I would like the whole thing to be read only access.

I have since moved my server to a ZFS-based system, but a few years ago I had a similar problem as you in trying to simply access content from multiple locations. One way that I addressed the issues in Windows was to use DFS. This doesn't address the permissions/readonly part, but DFS will allow you to present a share (referred to as a "namespace" as I recall), that within point to user-defined shares on the network. The user only has to go the UNC path "\\ServerName\DFSNameSpace" (ex., \\server\media).

For permission/readonly access, you could easily set it up where only one account (an administrator account) had write access with all other accounts having read only access.
tcs2tx is offline  
post #26 of 49 Old 10-26-2013, 05:56 AM
Senior Member
 
DraZtiK's Avatar
 
Join Date: Jun 2005
Posts: 277
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 15
I went down this road about 10 years ago and back then there were also not a lot of options and I tried like hell to work with junction points. Ultimately it forced me to go to Linux and I have not looked back. I run a headless Debian testing server and it took a bit but, I actually find it easier to manage via SSH and Webmin. It seems as if your set on Windows but if you do decide to keep your options open look into OpenZFS and/or AUFS, Snapraid also runs on Linux. I know people are recommending symlinks and mhddfs but AUFS is much faster and better suited for you than mhddfs. Good luck.
DraZtiK is offline  
post #27 of 49 Old 10-30-2013, 07:01 PM
Advanced Member
 
bulls's Avatar
 
Join Date: Sep 2005
Posts: 781
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 4 Post(s)
Liked: 22
Quote:
Originally Posted by StardogChampion View Post

Shares can be "pooled" on the clients using libraries if they're Windows 7/8 clients

didnt work for me... i just tried tthat. dumped some folders in a library and shared the library. this created separate shares for each folder frown.gif
bulls is offline  
post #28 of 49 Old 10-31-2013, 06:59 AM
Senior Member
 
Elpee's Avatar
 
Join Date: Dec 2007
Posts: 379
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 17
For those who are sitting on Windows OS, we don't have any other better solution for now? Damn, I'm being stuck here.
Elpee is offline  
post #29 of 49 Old 10-31-2013, 06:59 AM - Thread Starter
Newbie
 
JazzJackRabbit's Avatar
 
Join Date: Oct 2013
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 2 Post(s)
Liked: 10

I guess it's time for an update. In short most of my attempts to get this working failed.

 

Mounting Hard drives into the same folder: FAIL
- Can only mount a hard drive into an empty folder, mounting multiple hard drives into the same folder is not allowed.

 

Libraries: FAIL
- I can add multiple drives into the same library, but the view is not merged, i.e. if I have two drives, one with two folders music and movies and another drive with folders movies and series, I do not get a merged view showing me three folders music, movies, series at its root, The library does not combine/merge two drives together, instead when I go to the library view I see two sub entries one for each drive.

 

Windows DFS: FAIL
- I'm running Windows Server 2008R2 as the fileserver so I thought to try DFS as suggested by one of the posters above. I installed the file services role, installed DFS support, created standalone namespace, added folder, and inside that several folder targets, and this is really bizzare, but either I did sometime wrong, or DFS once again cannot "merge" folders. The bizzare part is that I added two folders to the DFS, and if I connected to the DFS share directly from the fileserver I would see folder/drive number one, if I connected from the client through the network, I would see contents of folder/drive number two, but never two of them combined. Like I said, I may have been doing something wrong, but I just couldn't make it work.

 

Symlinks: FAIL
- I used the snapraid pool command to generate symlinks for all of my files. In theory it would have been an acceptable solution, I would have to regenerate symlinks every time I added or removed something from the pool, but as long as I set it to run every night I don't think it would have been much of a problem. However, once again I just couldn't get it to work through the network. I could open symlinks on the server, but not on the client. I ran the fsutil behavior set SymlinkEvaluation L2L:1 R2R:1 L2R:1 R2L:1 utility in elevated command prompt on both the server and the client, but for some reason I still cannot access files through symlinks from the client. I even rebooted both the server and the client for good measure, and it still won't work. I may try this again on the off chance I screwed something up, but so far it does not look good :(

 

What worked.

The only thing that worked for me, PENDING FURTHER TESTING, is removing hidden and system attributes from PoolPart.guid folders. These PoolPart.guid folders is where DrivePool keeps all of its data. By default DrivePool makes those folders system and hidden, and SnapRAID by default ignores any hidden or system files or folders. I ran attrib -s -h command on every PoolPart.guid folder and now the SnapRAID seems to calculate parity for the PoolPart.guid folders. I say seems because it does not print out a warning message about ignoring those folders anymore when I run it, so I have to presume it calculates parity for anything inside those folders, however I have yet to test it. Later tonight I will try and delete a couple of files inside the PoolPart.guid folders and run the SnapRAID fix command to see if it restores the data. If it works, then I guess this is what I would have to go with.

 

A couple of extra notes.

 

I have asked the question about removing hidden and system attributes from PoolPart.guid folders on the DrivePool forums couple of days ago, and so far I have not received any meaningful answer. So far everything seems to be working fine, but it would have been nice to get some confirmation from the StableBit support that removing those attributes won't mess anything up.

 

I have yet to evaluate DriveBender. I really ought to, but I do not want to install Both DrivePool and DriveBender on the same machine. Does anyone know where does DriveBender keep the data when you write it to the pool?

JazzJackRabbit is offline  
post #30 of 49 Old 10-31-2013, 11:57 AM
AVS Special Member
 
Mrkazador's Avatar
 
Join Date: Jun 2005
Posts: 3,827
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 35 Post(s)
Liked: 248
Quote:
 Symlinks: FAIL

- I used the snapraid pool command to generate symlinks for all of my files. In theory it would have been an acceptable solution, I would have to regenerate symlinks every time I added or removed something from the pool, but as long as I set it to run every night I don't think it would have been much of a problem. However, once again I just couldn't get it to work through the network. I could open symlinks on the server, but not on the client. I ran the fsutil behavior set SymlinkEvaluation L2L:1 R2R:1 L2R:1 R2L:1 utility in elevated command prompt on both the server and the client, but for some reason I still cannot access files through symlinks from the client. I even rebooted both the server and the client for good measure, and it still won't work. I may try this again on the off chance I screwed something up, but so far it does not look good :(

 

 

I had a similar problem where my client systems couldn't access my shared Junction folders. What worked is adding "authenticated users" in the security tab of my shared folders.

 

Mrkazador is offline  
Reply Home Theater Computers

User Tag List

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off