[blackoper]

So I started playing around with a freenas 64 custom updated zfs build from this post: http://sourceforge.net/apps/phpbb/fr...tart=30#p27525 It includes zfs updated to version 13. So far it's pretty nice. I setup a zpool (raidz with 1 parity drive aka raid5z) with five 500gb drives. I'm still tuning it but it's very stable at 60MB/sec writes (havn't done read tests yet was loading back up a bunch of video files) on samba transfers, about 30 to 50% of the ram and uses about 50% of the cpu. I havn't gotten jumbo frames enabled yet on my win 7 box and I took my other linux pc down to get at the power supply. I plan on testing it with a linux box tomorrow hopefully and getting some transfer speeds numbers from an OS that knows how to use jumbo frames.


I also plan on testing out EON nas as well. It's just a lot more complex to setup than freenas.


Hardware specs:

Motherboard:Gigabyte G31M-ES2l Microatx

Gigabit chipset - Realtek RTL8111C

Processor: Intel E2140

Ram: 1GBx2 ddr2 random memory

Hard disks: Old 2GB laptop drive - may move over to a usb thumb drive but I don't see too much of a reason to since I had this sitting around

Add-in cards: Sil 3124 pci-e 2xsata for port multiplier.

Since it's kind of a big deal, the power usage for the NAS is below:

old 2gb laptop drive = 1.8 watts

multiple fans in case = 1 watt


without pci-e card and hard drives

picopsu-90 - 30 watts

antec earthwatts 380 - 36 watts

Thermaltake Toughpower 750 - 48 watts (no surprise there)


With pci-e card sata and hard drives on

picopsu -90 (N/A too much power)

antec earthwatts 380 - 89 watts

Thermaltake toughpower 750 - 98 watts


With pci-e sata card and hard drives spundown

antec earthwatts 380 - 41 watts

Thermaltake toughpower 750 - 50 watts


All 5 sata drives being written to at max speed over samba:

antec earthwatts 380 - 100 watts

Thermaltake toughpower - 104 watts


I may pick up one of the seasonic bronze 300 watt power supplies from newegg for $40. I need another power supply anyway and it should provide better effieciency than the earthwatts 380.


To do list:

Tune zfs and add some more ram to see if speeds increase.


Link aggregation with a second nic (for testing, I may not need it for my normal use)

Add in more drives. I've still got another five 2tb drives to add in (when they come in and another 5-in-3 sata enclosure to add), and two 750GB drives in raid0 I use for mythtv storage.


System note:

I was going to get a pinetrail board to be my nas but the supermicro board I like is about $180. It would take close to 8 years to pay for it in my area with the increased electrical savings. If the zotac board htat comes out in march with 6 sata ports, two pci-e slots and a mini-pci-e slot is cheap enough, I still may give that a go. Power rates here are 6.25 cents so 1 watt = $1 a year give or take.


2-19-10 - 9am - after tuning zfs and samba I have on average 80MB/s writes from a windows 7 box with jumbo frames off. Reads are: 85MB/s going to play with link aggregation next

 

[tux99]

Interesting report, thanks.


Why did you chose 500GB drives? I find 1TB-1.5TB is currently the sweet spot from a price/performance/proven-reliability/power-draw point of view for a raid setup.


With regards to pinetrail boards, I agree the SuperMicro is expensive (but still worth the price), but there is also the Gigabyte GA-D510UD coming out which seems very suitable (for up to 6 disks + IDE boot disk) while likely not costing much (haven't found prices for it yet, but I'd imagine less than US $100).

 

[blackoper]



Quote:
Originally Posted by tux99 /forum/post/18166174


Interesting report, thanks.


Why did you chose 500GB drives? I find 1TB-1.5TB is currently the sweet spot from a price/performance/proven-reliability/power-draw point of view for a raid setup.

Already had the 500GB drives. Got a good rate on them a few years ago

With spindown in use after 2 hours, there is no reason not to keep them. I'll just use them for backups or something.


I'll keep an eye out on that gigabyte board

 

[tux99]



Quote:
Originally Posted by blackoper /forum/post/18166327


Already had the 500GB drives. Got a good rate on them a few years ago

Well, if you had them already then it makes perfect sense.


How is that 5 in 3 cage with regards to heat?

I have always wondered if those cages don't make the hard disks too hot as they are packed quite closely together.

It would be great if you could provide some hddtemp outputs, measured for example after running a 30 minute continuous read or write to all 5 disks at the same time (basically from/to the raid device).

 

[mythmaster]

Awesome! But why SMB instead of iSCSI ?


EDIT: Nevermind. Guess you'd need some kind of clustering filesystem for multiple initiators.

 

[blackoper]



Quote:
Originally Posted by tux99 /forum/post/18166404


Well, if you had them already then it makes perfect sense.


How is that 5 in 3 cage with regards to heat?

I have always wondered if those cages don't make the hard disks too hot as they are packed quite closely together.

It would be great if you could provide some hddtemp outputs, measured for example after running a 30 minute continuous read or write to all 5 disks at the same time (basically from/to the raid device).

I've noticed the drives stay cooler in the 5-in-3 case than they do sitting by themselves. I believe this is mainly due to the cooling fan that pulls air through.


35 min of constant read and write from multiple computer and smart indicates drive tempsin celsius of:

drive 1: 34

drive 2: 33

drive 3: 32

drive 4: 31

drive 5: 29


34 celsius = 93.2 degrees F

 

[mythmaster]

A couple of more questions for you because I'll be doing something similar:


* Isn't NFS faster than SMB? and

* Can't Windows clients use NFS? http://support.microsoft.com/kb/324055


The Windows client that I'll be using is 2008 Server because I'll want to publish Windows apps through terminal services, but I'll want it to have access to the share so it can dump media files there (e.g., from AnyDVD).


I just don't want to use SMB unless I absolutely have to.

 

[tux99]



Quote:
Originally Posted by blackoper /forum/post/18172402


I've noticed the drives stay cooler in the 5-in-3 case than they do sitting by themselves. I believe this is mainly due to the cooling fan that pulls air through.


35 min of constant read and write from multiple computer and smart indicates drive tempsin celsius of:

drive 1: 34

drive 2: 33

drive 3: 32

drive 4: 31

drive 5: 29


34 celsius = 93.2 degrees F

Those temperatures are excellent, thanks for taking the time to test that, I wonder if you could do the same test with the fan deactivated too to see how much difference it makes?

 

[blackoper]



Quote:
Originally Posted by mythmaster /forum/post/18172613


A couple of more questions for you because I'll be doing something similar:


* Isn't NFS faster than SMB? and

* Can't Windows clients use NFS? http://support.microsoft.com/kb/324055


The Windows client that I'll be using is 2008 Server because I'll want to publish Windows apps through terminal services, but I'll want it to have access to the share so it can dump media files there (e.g., from AnyDVD).


I just don't want to use SMB unless I absolutely have to.

I have never tried NFS on a windows system. I usually just run samba and call it a day. Performance of nfs has always been better when tuned, but with my usage samba makes more sense.


To setup nfs on a win 7 box: http://sagehacks.wordpress.com/2009/...der-windows-7/


It looks like you can get NFS working for 2008 server but it's a lot more complicated. I don't really see the reason unless you have a weaker processor than I do. I'm getting about 800mbits average with samba and zfs so. Nfs may boost that by 50-100mbits but the ease of use would be harder. The exception would be if you have multiple raid arrays and are using link aggregation with multiple gigabit cards. Then the lower overhead of nfs may make sense.


Tux - I'll give it a go and see what the temps indicate


Also I tried sitting up link aggregation following this guide: http://sourceforge.net/apps/phpbb/fr...772864b424b507 and I couldn't get it to work. I'm running a dell 2716 switch that supports it and a linksys 150n with ddwrt. I couldn't get the link to use an ip that I could connect to. I removed it as it's not required in my install. I may try again later

 

[mythmaster]

Thank you, blackoper. It will be running in a VM, so I want the overhead to be as minimal as possible. Also, "ease of use" doesn't concern me because I have time to kill. Thanks again.

 

[blackoper]

So as another observation, zfs raid is more fault tolerant then mdadm was. I had a drive in a xfs raid5 that kept being failed/kicked out of the raid pool. Manufacturer's testing tools indicated drive was fine. Zfs vers. 13 has had no issues with it at all.

 

[drkdiggler]

Would you mind posting some information on how tuned your Samba setup? I've only been able to get Samba (on Ubuntu 8.04 LTS) up to about 25 MB/s even though the array can run at 160 MB/s. Thanks!

 

[blackoper]



Quote:
Originally Posted by drkdiggler /forum/post/18187436


Would you mind posting some information on how tuned your Samba setup? I've only been able to get Samba (on Ubuntu 8.04 LTS) up to about 25 MB/s even though the array can run at 160 MB/s. Thanks!

First off you should test how fast your speeds are using ftp file transfer to confirm it's the samba/cifs sharing that is the issue. If your ftp transfer is the same speed it could be the network/wiring/network card driver.


I added a lot of the tweaks from this page: http://learnedbyerror.blogspot.com/2...une-er-up.html


The parameters added to the samba config are:

Quote:
max xmit = 65535 socket options = TCP_NODELAY IPTOS_LOWDELAY SO_SNDBUF=65535 SO_RCVBUF=65535

along with the changes above for samba I'm not sure what the parameter is for this. It's a checkbox in freenas that I checked:

Enable large read/write

Use the new 64k streaming read and write varient SMB requests introduced with Windows 2000.


Freenas specific tuning:

besides the samba changes above, for anyone wanting to run freenas the only changes I have made over a base install is in the System|Advanced menu I have Tuning checked and for my loader.conf it looks like below. I have 2GB of ram, if you have less you'll need to adjust the kmem sizes:

Quote:
mfsroot_load="YES"

mfsroot_type="mfs_root"

mfsroot_name="/mfsroot"

hw.est.msr_info="0"

autoboot_delay="5"

vm.kmem_size_max="1536M"

vm.kmem_size="1536M"

vfs.zfs.prefetch_disable=1

 

[Troubleshooter]

Not sure that bonding your nic's will do much for you with just a single thread going on. You can go ahead and throw them into mode 5 or 6 since ya got the ports, why not, but with Samba I doubt you'll see anything unless you do multiple streams. I believe SMB2 can use multiple streams but I haven't played with it.


Now as an alternative to Samba, it may be worth checking out Likewise. It's pretty damn lightweight and supports SMB2. There's an opensource version as well. I've been working with it (at work) and it's showing potential.

-Trouble

 

[mythmaster]

Is this the Likewise that you're talking about, Troubleshooter? http://www.likewise.com/download/index.php If not, can you post a link or two?


Also, are you the same blackoper that I remember from the XBMC forum when we were testing VDPAU with motd2k? That guy was dropping code like crazy -- it was unbelievable how fast he got it working. I was so impressed that I gave them money even though I was unemployed.

 

[Troubleshooter]

Yes, thats the one - you can go to

http://www.likewiseopen.org/


The 'fileserver' part is what is interesting. The rest of it is for AD integration which is of great interest to the corporate side of things.


-Trouble

 

[blackoper]



Quote:
Originally Posted by mythmaster /forum/post/18197431



Also, are you the same blackoper that I remember from the XBMC forum when we were testing VDPAU with motd2k? That guy was dropping code like crazy -- it was unbelievable how fast he got it working. I was so impressed that I gave them money even though I was unemployed.

My username in XBMC forums is blackoper as well and I was involved with the vdpau testing when I got an 8400 gs a ways back so I'd say it is likely.