AVS › AVS Forum › Video Components › Home Theater Computers › HTPC - Linux Chat › External Array Recommendations
New Posts  All Forums:Forum Nav:

External Array Recommendations - Page 3

post #61 of 102
10/16/09 at 6:20am
Quote:
Originally Posted by quantumstate View Post

[...]
Then test it:
# time dd if={somelarge}.iso of=/dev/null bs=256k
[...]

Thanks, the best I get is 14.8MB/s, which seems good to me.
The write speed is more interesting actually as it's SLC so should be much faster than most USB sticks. I will test that soon too.
Reply
Reply

AVS Top Picks

post #62 of 102
10/16/09 at 8:08am
Thread Starter 
OK, I guess you are reading at the max cell access speed. No positioning or latency time to eliminate.

SanDisk Extreme is rated at "up to" 30MB/s read and SanDisk Ultra is rated at 15MB/s read and 9MB/s write speeds. In practice, my SanDisk Cruzer Micro 4GB tests at 23.2 MB/s at the default readahead cache of 256KB. Not sure whether it's considered the Extreme or Ultra.

My freakin' RAID10 array, with 4096KB readahead on each sd drive and (an automatic) 8192KB on the md device tests at 1.6GB/s! Good news, and makes it worth it.
Reply
Reply
post #63 of 102
10/16/09 at 10:14am
Thread Starter 
BAH! It tricked me. Even though I'd run the test before on the USB drive, and I'd renamed the file, it didn't fool the cache. When I rebooted I got the normal numbers, not no gigabyte. My best readahead cache size for the array with WD 2TB drives is 4096.

I also ran tests on the new Scorpio 2.5" 7200rpm boot drive:
512 84.6MB/s
1024 84.1
2048 83.9
4096 83.4
8192 83.4

Just as well because a smaller cache is better, as a boot drive will be dealing with small files, as opposed to the array which does videos. Heck, I'd better keep testing down.
Reply
Reply
post #64 of 102
10/16/09 at 10:40am
I was going to say, looked a little too good to be true.

Perhaps I missed it in the thread, but why the choice of USB over a SATA or IDE to CF adapter?
Reply
Reply
post #65 of 102
10/16/09 at 10:47am
Thread Starter 
Haven't selected USB yet, but is is more versatile than DOM. Didn't think of a SATA to CF adapter.
Reply
Reply
post #66 of 102
10/16/09 at 12:40pm
Quote:
Originally Posted by drkdiggler View Post

I was going to say, looked a little too good to be true.

Perhaps I missed it in the thread, but why the choice of USB over a SATA or IDE to CF adapter?

Simply because 40$ for a 8GB SLC flash device is cheap, a 8GB SLC CF card costs considerably more. The drawback is that this SLC USB stick is slower than typical SLC CF cards.

The Transcend Industrial CF card range is SLC and very well suited (I have a 8GB in my Linux home router/gateway), but quite a bit more expensive.
Reply
Reply
post #67 of 102
10/20/09 at 4:58pm
Thread Starter 
Ah sh*t, I've made a mistake. Whereas I was looking for an industrial-grade enclosure built fairly strong for rough environments, I just got this Qbox4 and it is very pretty. Lots of acrylic and very aesthetically appealing for gameboyz, it's not what I was looking for. Lots of crevices for dust, and fancy-pants goo-ga's. Well, I guess it's what I have.

Quality isn't terrible, it's just more -+*Style*+- than suits me. I guess I'll put a plastic bag over it to keep out the dust.
Reply
Reply
post #68 of 102
10/21/09 at 1:20am
Quote:
Originally Posted by quantumstate View Post

Ah sh*t, I've made a mistake. Whereas I was looking for an industrial-grade enclosure built fairly strong for rough environments, I just got this Qbox4 and it is very pretty. Lots of acrylic and very aesthetically appealing for gameboyz, it's not what I was looking for. Lots of crevices for dust, and fancy-pants goo-ga's. Well, I guess it's what I have.

Quality isn't terrible, it's just more -+*Style*+- than suits me. I guess I'll put a plastic bag over it to keep out the dust.

Sorry to hear that. you went through so many different ideas in this thread that it's hard to tell what you ultimately wanted.
I only suggested that case as a much less expensive do-it-yourself alternative to the Qnap NAS you were considering (which certainly isn't industrial grade and dust-proof either).
I have never tried it, but I read once that the trick to avoid dust is to make the case fans blow into the case instead of out of the case and then to add a filter in front of the case fans.
When the fans blow into the case, then it doesn't matter that the case is dust-proof as the air inside will try to get out again through the gaps, not be sucked in, so no dust comes in other than through the fans which have filters.
Reply
Reply
post #69 of 102
10/21/09 at 4:56am
Thread Starter 
Not your fault tux, I simply ordered the best enclosure I could find. Decided you're right that a proprietary solution is no good. The need for an ITX box with open front hasn't been recognized yet, and this is just about the only choice.

This netgear wndr3700 router is built cheaply, even as it's about the most expensive there is. It'd better perform better than the dlink dir-825 without hangs...
Reply
Reply
post #70 of 102
10/21/09 at 6:24am
tux, the suggestion about having all fans run as intake would probably be helpful. I was a semiconductor engineer in my previous life; the cleanrooms were run at higher pressure than the surrounding building to help prevent dust/dirt from entering. One thing to keep in mind is that your power supply probably blows out, so you may have to disassemble it to change the direction of the fan.
Reply
Reply
post #71 of 102
10/21/09 at 6:33am
Thread Starter 
Yes I am suspicioning that the PSU fan will be the only one. I'll check its direction when I've decided on a mobo. The sides of the case are almost all vent, and there would be no plenum effect no matter where I put case fans.
Reply
Reply
post #72 of 102
10/25/09 at 6:01am
Thread Starter 
As this severe filesystem corruption has happened to me twice now, with BTRFS and JFS, I am suspicioning the mobo. I have a very good power-conditioning transformer and battery backup on the system, so it wasn't a power glitch. It could be a mobo hardware failure, or it could be a design problem; but whatever, I am considering to replace it with a Gigabyte something-or-other.

No, as soon as I reinstalled jfsutils the array began working again, as if it was maybe missing a library for fsck.jfs or something.

Now that I believe BTRFS was not the reason for my prior loss of data, and that I am constructing a remote storage server, I think I'll try BTRFS again at least on one system. Checksumming is just so desirable, and I am concerned with the layering in ZFS-FUSE. Also, I'm not so sure about mdadm anymore as it didn't save me from this latest JFS filesystem corruption.

So I may use some function of BTRFS for syncing the two systems (HTPC & SAN), or maybe an rsync and checksumming function and let the SAN system sleep most of the time. This means that I'll need a mobo for it that does wake-on-LAN successfully and automatically, and this would pretty much eliminate the dust problem, being in the garage.
Reply
Reply
post #73 of 102
10/25/09 at 4:55pm
Quote:
Originally Posted by quantumstate View Post

As this severe filesystem corruption has happened to me twice now, with BTRFS and JFS,
No, as soon as I reinstalled jfsutils the array began working again, as if it was maybe missing a library for fsck.jfs or something.

Ok I think I know what happened (not what caused the freeze but the superblock error you saw at boot afterwards).
Did the error you saw look like this?
Code:
fsck.ext3: Invalid argument while trying to open /dev/md0
/dev/md0:
The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193
Reply
Reply
post #74 of 102
10/25/09 at 8:31pm
Thread Starter 
Close. It could not mount the JFS filesystem because it could not find a valid superblock on md2.
Reply
Reply
post #75 of 102
10/25/09 at 9:25pm
Quote:
Originally Posted by quantumstate View Post

Close. It could not mount the JFS filesystem because it could not find a valid superblock on md2.

Ok forget the fact that my error message is for ext3, that's because my test filesystem was ext3 not jfs, but the cause is the same.

Basically that error happens when the underlying raid device hasn't been assembled yet. For various reasons it can happen that during boot when the OS is trying to check all filesystems that have a non-zero sixth field value in /etc/fstab (see man fstab) the raid isn't ready yet and therefore the necessary md0 (or in your case md2) device doesn't exist yet.

In other words, there was absolutely no filesystem corruption on your raid, the filesystem has never suffered any problems, only the raid was slow to reassemble (probably a consequence the freeze, ie unclean shutdown).
This is also supported by the fact that you didn't have to do any fsck to get the jfs filesystem mounted again.

Of course all this doesn't help finding the cause of your system freeze, but it should restore your confidence in md raid and JFS.

One thing that you might wanna investigate is why you missed the jfsutils package, that had probably been uninstalled by some debian update with dodgy dependencies (I seem to recall you are using debian-experimental?).
Reply
Reply
post #76 of 102
10/26/09 at 5:36am
Thread Starter 
The array wasn't slow to assemble, as I'd left it in the half-booted state while I went to work. And yes maybe jfsutils had been deinstalled by an apt-get malfunction. It's the only plausible explanation.

But also understand what this episode implies: any filesystem glitch in the RAID10 volume destroys everything. I thought that building JFS on the array meant building it independently on the two mirror sides, but that doesn't seem to be the case. When I tried to mount the sd drives independently, mount didn't understand this RAIDvolume filesystem. And when I pulled the plug on each side respectively and rebooted I still could not see my data. True, in this case it was apparently a missing JFS lib, but what if the filesystem table got a glitch? No more nothin'. This was a shock to me. I thought I had two independent sides.

So I've lost confidence in mdadm, and am exploring other options. ATM I'm thinking the garage storage server can act as a backup, and I can rsync the HTPC array to that, with the --checksum option to confirm every file is truly backed up. When not in use the storage array could sleep. This would be an independent copy of all my hard-won data, in case of fire, fat-finger, or theft. I don't need the data instantly, as with RAID10.
Reply
Reply
post #77 of 102
10/26/09 at 6:03am
Thread Starter 
You know, I was worried that my main HTPC would be overloaded, with Myth commercial flagging, transcoding, and storage server operations like ssh encryption and checksumming.

But clearly storage server operations should be carried out by the storage server. This'll be a little qbox-4 mini-ITX system in the garage in case of fire or theft. So I could iSCSI the HTPC's RAID0 volume to the storage server, hopefully through an SSH tunnel, and the storage server could merrily do the rsync, checksumming, and whatever operations, while the HTPC does its proper jobs. When not in use the storage server would sleep. Sure, the storage server will be just an Atom or Geode, but as long as its operations don't take more than 24 hours it's OK.
Reply
Reply
post #78 of 102
10/26/09 at 6:27am
Quote:
Originally Posted by quantumstate View Post

The array wasn't slow to assemble, as I'd left it in the half-booted state while I went to work.

I'm not sure what you mean by that, but I'm pretty sure that if you had done a cat /proc/mdstat right after getting the "invalid superblock" error you would have found that the raid was not assembled.

Quote:
Originally Posted by quantumstate View Post

But also understand what this episode implies: any filesystem glitch in the RAID10 volume destroys everything. I thought that building JFS on the array meant building it independently on the two mirror sides, but that doesn't seem to be the case. When I tried to mount the sd drives independently, mount didn't understand this RAIDvolume filesystem.

RAID10 is not the same as RAID1 (mirror). With RAID1 you can mount each half of the mirror separately (I have done that in the past), with RAID10 you can't as the data on each half is striped.
You need to use RAID1 to have the kind of failure tollerance you want.

Quote:
Originally Posted by quantumstate View Post

True, in this case it was apparently a missing JFS lib, but what if the filesystem table got a glitch?

It wasn't a missing JFS lib as the jfsutils package doesn't contain any libs, only utilities. You can mount a JFS filesystem without jfsutils installed, I just tried it here as long as it's clean. In your case it obviously wasn't clean as the box hadn't been shutdown cleanly so it required the jfs.fsck to reset it to clean.
Read up the following article to understand why JFS survives system crashes without any problem better than other filesystems:
http://www.ibm.com/developerworks/library/l-jfs.html

If the filesystem table (or rather superblock) got a glitch as you say, then you have a problem with any filesystem, but that doesn't happen easily as the superblock is stored in multiple copies.
Reply
Reply
post #79 of 102
10/26/09 at 7:08am
Thread Starter 
{cough}
Moving on.
Reply
Reply
post #80 of 102
10/26/09 at 7:12am
Quote:
Originally Posted by quantumstate View Post

{cough}
Moving on.

I was just trying to make you (and anyone else reading this) understand better what happened, but if you're not interested then I'm moving on too...
Reply
Reply
post #81 of 102
10/26/09 at 7:17am
So to jumpstart that moving on :-)

Have you given any considerations on whether you want to use wake-on-LAN vs. using the RTC to wakeup the machine? If so, I'd like to hear your thoughts. WOL is nice because the machine can wake up at any point if you want to run a backup for a certain reason, as opposed to waiting for a scheduled event. Also, how are going to deal with the time delay between sending the wakeup request and when all of the services are up and available? Simple 2 minute pause, or will you poll them until they are available?
Reply
Reply
post #82 of 102
10/26/09 at 8:25pm
Thread Starter 
Just today solidified that the storage server should perform all backup operations, not the HTPC, but haven't sorted out how yet. It should wake at a set time and do its work, accessing the HTPC's array through iSCSI. And it must not take more than 24 hours for a cycle.

I know Myth machines can go to sleep and use RTC for wake events, but I haven't studied that in detail as my machine is usually recording Dish. However I suspect something similar could be used for rsnapshot or similar. The storage server will wake itself and initiate its work, so timing after wake isn't a concern

I'm working on many other things, but am striving for a best practice top-flight system here. I generally post progress here and welcome input.

And thanks tux, but that was a distressing episode that I'd just as soon avoid. I'm going to run memory tests, but expect to find nothing and will probably replace the mobo with a newer Gigabyte. Historically I haven't trusted Gigabyte and do Asus, but they are not allowed to corrupt my disks whether it's a hardware problem or not. OK maybe it was that jfsutils was deinstalled, but the result was still distressing.
Reply
Reply
post #83 of 102
11/4/09 at 7:55am
Thread Starter 
I think I've identified all worthy candidates for mobo in my storage server to the below. The absolute minimum standard is at least 3 internal SATA ports and at least 1 GbE port.

I was hoping MSI had something good, but not one of their mobos has more than 2 SATA ports. This was a complete surprise and disappointment.

ASUS AT3N7A-I (review)
It's ION-based, which I don't really need because it will be headless, although it does come with CPU reducing that cost. But will that be enough of a benefit? Takes DDR2, which I prefer for now. $129.87

Gigabyte GA-2AIEV-RH
It's AMD which is desirable, and has the AM2+ socket which will take some of the newer AM3 CPUs. Has 6 SATA ports (!), and PCI-E 2.0 x16 rather than PCI as is usually the case. (and was dictated by Intel) Takes DDR2, which I prefer for now. Eh, Froogle, NewEgg, and eBay never heard of it, so can't pin down a price yet.

Gigabyte GA-6KIEH-RH
This takes the Penryn /Merom processors, and although I don't know what wattage range these are, I'll bet they're low. Has 5 SATA ports (Silicon Image, very good), but PCI, which may not be a disadvantage as most surveillance camera cards are PCI, although it's a ghastly 32-Bit/33MHz. But look at this, it also has a mini card slot (PCI-E x1/ USB 2.0) and a mini PCI slot (PCI 33Mhz), for other uses like a wifi card. Also an IDE connector, 2 Serial connectors, 2 GbE (WOL, PXE, AFT, ALB , SLA, 802.3ad, SFT), and DDR2, but Intel IGP which I will hardly use. An astonishing $178 without CPU or boot device.
Reply
Reply
post #84 of 102
11/4/09 at 1:01pm
if you are looking into an atom board this one is pretty nice:
http://www.newegg.com/Product/Produc...82E16813135247

pci-e 16x, pci-e 1x and a pci slot. Gigabit ethernet and 3 sata ports. The Atheros AR8131M lan chipset is supported in linux and from what I can tell freebsd 8 and opensolaris's/nexenta's newest version. If I get this I plan on throwing on freebsd 8 and using it as a zfs server. Power usage should be in the 30 watt range. With the pci-e slots and port multipliers you could run a lot of drives off this board (30+) I currently run about 12 drives and plan on expanding so the extra port multipliers are important to me.

I know it's an ecs, but the reviews and blogs seem to indicate the board is holding up well and not having any issues. The quality of their atom builds appears to be better than the normal stuff they do.
Reply
Reply
post #85 of 102
11/4/09 at 1:14pm
Quote:
Originally Posted by blackoper View Post

if you are looking into an atom board this one is pretty nice:
http://www.newegg.com/Product/Produc...82E16813135247

That's a mATX board, quantumstate has a mini-ITX case to fill.

quantumstate of the 3 boards you mentioned the AMD board is probably best, it will consume more power than the other two, but Socket P processors are VERY expensive (unless you can find a second hand one on ebay for cheap) and the Atom+ION board doesn't have enough SATA ports.
Else you could get a 2 or 4 port PCI SATA controller card with the Atom board, but it might affect performance slightly because of the PCI bottleneck (might not be noticeable in practice).

edit: actually thinking of it, a 2 port PCI SATA controller won't be a bottleneck (133MB/s) and with the 3 on-board SATA connectors you have the 5 ports you need (you could use a IDE to CF adapter to boot from).
So an Atom board with 3 SATA on-board and a PCI slot with a 2 port PCI SATA adapter is probably the best solution for cost and power consumption.
The AMD board will likely idle at 70-80 Watts with a 45W TDP cpu and 5x 5400rpm 3.5" hard disks, while the Atom board will be around 50 Watts or less (always with 5 hard disks).
Reply
Reply
post #86 of 102
11/4/09 at 1:32pm
whoops, I thought I read over everything before I posted and just missed the itx part. A little destracted since I'm busy at work and multitasking with the forums stuff
Reply
Reply
post #87 of 102
11/5/09 at 5:17am
Thread Starter 
Can't occupy the pci slot, as that's reserved for a camera board, however I could put on a port multiplier (adding to the cost).

Each of the boards has one problem or another I that don't like. The AMD board doesn't have a PATA port so I can connect a CDROM to load OS, and it doesn't seem to be available as no one has a price.
Reply
Reply
post #88 of 102
11/5/09 at 6:07am
Are you going to be running the OS off of the CDROM all of the time or are you just installing once? Why not install from USB or via a network install?
Reply
Reply
post #89 of 102
11/5/09 at 8:57am
Thread Starter 
Will be running the OS off of some sort of flash, so planning install media.

Have only one USB stick ATM, and that's being used. Assuming the BIOS can boot from the network, I didn't know a direct network install was possible. Only knew of the skeleton boot CD for network. Concerned for security with a direct net install; want shorewall up all the time. Besides, I'll bet any Debian Testing net install would be Gnome, not KDE.
Reply
Reply
post #90 of 102
11/5/09 at 9:33am
Check it out (if memory serves correctly, you are a Debian guy):

http://wiki.debian.org/NetworkBooting
Reply
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: HTPC - Linux Chat

AVS Top Picks

AVS › AVS Forum › Video Components › Home Theater Computers › HTPC - Linux Chat › External Array Recommendations