I am quickly running out of space on my 2TB RAID10 array, and there's no more internal mounting in my case. Seems like the next logical step is an external array.

Now; my setup is first class, and I want to keep all parts that way. Ideally the array would carry like 6-10 drives, be 19" wide, and have a black high-grade aluminum look. My case is the Silverstone GD02-MT (http://www.silverstonetek.com/products/p_contents.php?pno=gd02mt&area=).

Needless to say it should be SATA not USB or firewire, and ideally it should externally expose the drives for hot-swap. I should think there are some great server solutions out there.

Silverstone has a pretty good one (http://www.silverstonetek.com/products/p_contents.php?pno=ds351&area=usa), but it's only for 5 drives and doesn't have all I want. Says it does all the RAIDs, but I presume that's in hardware. I need to RAID10 with the MD system in software.

Any suggestions?

I am quickly running out of space on my 2TB RAID10 array, and there's no more internal mounting in my case. Seems like the next logical step is an external array.

Now; my setup is first class, and I want to keep all parts that way. Ideally the array would carry like 6-10 drives, be 19" wide, and have a black high-grade aluminum look. My case is the Silverstone GD02-MT (http://www.silverstonetek.com/products/p_contents.php?pno=gd02mt&area=).

Needless to say it should be SATA not USB or firewire, and ideally it should externally expose the drives for hot-swap. I should think there are some great server solutions out there.

Silverstone has a pretty good one (http://www.silverstonetek.com/products/p_contents.php?pno=ds351&area=usa), but it's only for 5 drives and doesn't have all I want. Says it does all the RAIDs, but I presume that's in hardware. I need to RAID10 with the MD system in software.

Any suggestions?

Take a look here. http://www.scsi4me.com/sata-serial-ata-external-enclosure.html

Oooo, I like (http://www.enhance-tech.com/products/diskarrays/R8_index.html).

After a good bit of research, the only HTPC-grade external storage is the Silverstone, and there are reports that it's problematic. I've just about decided that I'll have to go with a rackmount of some sort.

One thing that's not clear is whether you can daisy-chain e-SATA?

Whelp, here's an update on the lawsuit I was prosecuting against the company which sold me the propane system to convert my car, and its manufacturer, as some here know. Yesterday after two years of research, study, and legal work in my spare time paid off. I won my lawsuit for denial of warranty on the propane injection system for my car.

For a year I tried and tried to get the simple part which had failed in my system and was causing chronic alarms and fault-over to gasoline at every single stop sign. But they delayed, and deferred, and made countless empty promises, and hoped I'd get lost.

So when I wised up after a year I filed suit in Washington =Superior= Court asking for a refund of everything, a new engine, peace in the Middle East, and that defendants be taken out and executed. They hired a New York lawyer to harass and further obfuscate, obstruct, and delay... but I took advantage of his errors and oversights and got him banned from the case, as 'not licensed to practice law in the State of Washington'. As further punishment the judge struck all their pleadings and defenses, including their original answer which is crucial to their defense.

I then noticed them to appear at a deposition (colloquially called the 'nuclear option') so I could formally question them... which they failed to do; not too surprising since one company is in New York and the other is in The Netherlands, but nevertheless failure to appear at deposition is a serious offense to the court. So I then filed a motion asking the Court to order them to appear and it did order them to appear at two second scheduled depositions (one for each defendant), which of course they each failed to do. (for a $5,000 case? Why would they?) So I filed a Motion for Sanctions, which the Court granted and also by the way found them in Contempt of Court for failing to appear at the Ordered deposition.

Well, it wasn't quite over. I then sent each of the defendants Requests for Admissions, asking them to admit the warranty, their full responsibility, product liability, failure to honor my valid warranty, responsibility for the Kennedy assassination, etc, and lo and behold their New York attorney dared to answer and sign them. He had earlier been ordered by the Court to no longer attempt to represent them, or else, remember? So technically he was in Contempt of Court, but I had bigger fish to fry. I asked that the Court deem all my Requests for Admission, to be admitted to by defendants, in full and conclusively as provided in Washington state law, since the New York attorney was not authorized to answer them and their 30 days were up since he sent them back at the last minute as usual. (I was counting on this) The Court granted my request on July 2nd, obliterating their case and deeming that defendants had admitted to everything plus the kitchen sink.

Now there was absolutely nothing left in question in the case and I was pretty sure I had it won, so I filed a Motion for Summary Judgment. I had to wait two months for it to be heard, as the docket is backed up, so it was scheduled for yesterday. However this past Monday I got a phone call from a local attorney saying he had just been appointed to represent defendants. Bad news, as I thought they would just not show and I'd win by default. Well he came out with knives blazing; Motion to Strike my summary judgment on numerous grounds, Declaration in support of same, Motion for Continuance, Motion to Extend Time, Notice for Hearing, etc. I studied all his filings and didn't see how they'd affect anything for anything. Their case was already dead, but I filed an eviscerating Response just the same.

So yesterday we both showed up for the trial. Right off the bat the judge knocked off all his filings, citing abuse of process in their waiting until the 11th hour to do all this. The attorney was visibly shocked. Then the judge considered my Motion for Summary Judgment. She asked for my testimony and I basically read off the important points of my Motion. However I had not put in a 'magic sentence' certifying that all my statements in my Declaration were true and correct, but I requested to be put under oath and certify right then, and she allowed it. She asked for his rebuttal, and he was clearly not ready but made a valiant attempt to shoot down my case; he had expected to get his motions granted I guess, and hadn't even considered this possibility. He is a good attorney as they go, but missed alot of things I would have done for the sake of rigor. Looked like a routine hourly type work-a-day dude to me, not no top-gun ferocious litigator like I'd battled in the past. I rebutted, and the judge didn't take much time to render her decision.

She used terms like, "It's time to put this case out of its misery", and the "long-suffering Mr. {me}" in reference to my dealings with defendants, and "he is a credit to pro se's". She granted me a judgment of $15,480.00 plus costs and post-judgment interest accumulating at 12%/annum. Unfortunately I hadn't had time to research the law on treble damages, or I would have probably gotten that too. The attorney then questioned the judge about whether she has jurisdiction, the fact that the defendants are out-of-state and do not do business here, and that they hadn't sold me the system, clearly angling for grounds for appeal, but I countered that all these questions had been deemed admitted conclusively in my Requests for Admission... completely destroying any hope he had.

We both thanked the judge and she retired to her chambers. I waited around while the attorney got his stuff together, then asked whether we have anything further to talk about. He was clearly angry (I guess for losing such a case so badly to a lowly pro se [self-represented plaintiff]... little does he know who I am) and said no, that he has to confer with his clients to see whether they want a motion for reconsideration or to appeal... so I left. I was a gracious winner and didn't intend to embarrass him.

I am pretty sure they won't make further filings, given their past abuse of process and reputation with the Court, so I might get paid soon. If not, I get to learn about the Collections process in a court at this high of a level, and in The Netherlands. defendants have 30 days to appeal, and if I hear nothing I begin collections with a Demand Letter to each defendant. They have been found liable jointly and severally, meaning that I can look to either one for all of it, or both. I don't care how they divide up responsibility. I'd like to listen in on that fight though...

Now see; If they had just sent me the blasted $95 part, I would have fixed it myself. Or if they had actually followed through with one of the four times they'd promised to fix my car but failed, this all would never have happened. Think they'll learn a lesson here and start treating other warranty claims more seriously? (Naaah) A few months ago I told them it looks like 'there must be something in the water' over there in New York, with all these broken promises...

Luckily my specialty and avocation is punishing businesses who routinely rip people off who are less able to defend themselves, insurance companies especially. Think an unregulated market actually works? Check out this story:
California's Real Death Panels: Insurers Deny 21% of Claims (http://www.reuters.com/article/wtMostRead/idUS202570+02-Sep-2009+PRN20090902)

I now have the part on order from the Dutch manufacturer's rep in Greece. Cost me 38 Euros (~$57), but it remains to be seen whether they have their act together well enough to send it to me. No other manufacturer's rep in the world would respond to my request except Portugal, which wants 210 Euros for the same part! Pffffff...

One thing that's not clear is whether you can daisy-chain e-SATA?

Pretty sure that you can't. Congrats on your win, btw. :cool:

Thanks.

I'm gathering the same. Seems there are certain ways external storage is connected:
- In consumer systems firewire and USB
- Enterprise, SAS, Infiniband, multichannel, and SCSI
- New way, e-SATA.

I want e-SATA as it's already built in to my mobo.

Also I find that there is often a motherboard inside the DAS box, which mediates transfers and implements RAID in software. This is the case with most RAID DAS boxen. I don't want this. I want as direct a connexion as possible with only an e-SATA to SATA port replicator inside.

So after combing 10 pages of a Clusty search, I guess I've narrowed my solution-set to these:
http://www.sansdigital.com/accustor/index.php
http://www.enhance-tech.com/products/diskarrays/R8_index.html
http://www.enhance-tech.com/products/ultrastor/RS8_index.html
http://www.aicipc.com/Category.aspx?ref=RSC2U
http://www.iostor.com/index.asp?PageAction=VIEWCATS&Category=37
http://www.istarusa.com/rackmount_chassis/products.aspx
http://www.twinipc.com/ProductDetail.aspx?ref=RSA-2ED2-2
http://www.twinipc.com/Category.aspx?ref=RSC2U
http://www.bsicomputer.com/new/storage/rs2u8-ms/rs2u8-ms_intro.htm

Mainly collected names for 2U chassis' up to this point and I haven't filtered them yet for e-SATA nor 80+ power supply. I'd like run/fail lights, silence, removable rack ears, and hot-swap but not mandatory.

Looks like most of those have a mini-SAS host connection which would give you the same performance as eSATA, but you'd have to get a card (http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&N=40000410%201193320805&Description=mini%20sas&bop=And&ActiveSearchResult=True&Order=PRICE), obviously. Seems like one of them supported firewire/eSATA, though.

I could be wrong, I just glanced through them.

Let us know what you decide to go with and why, please. I'll be doing the same with my next server build. It'll be a while, but I'm going all-out with a rackmount and 4 CPU board and ext storage so I can start experimenting with some app hosting. Dom0 will prolly be NetBSD since they just reworked their threading model from scratch.

One thing that I'll do, and you may want to consider this, also, is run OpenSolaris in a headless VM as an iSCSI target so I can use ZFS on my ext storage.

Yup. The only thing that comes even close is this (http://www.sansdigital.com/elitestor/index.php), but not enough drives, and it has an individual e-SATA connector for each drive, rather than an e-SATA-to-SATA2 port multiplier. What, am I going to have to jerry-rig something? Are we doing New Science(tm)?

If I just get the enclosure, I need to make sure it has a hot-plug SATA2 backplane and ~300w PSU for 8 drives, but does hot-plug also impact the port multiplier? Who knows.

As to OS, I'm pretty happy with Debian Testing. Has some freakin' annoyances, but it's safe.

Headed for Friday Harbor this evening for a couple days. (work)

Port Multipliers (http://www.sata-io.org/technology/port_multipliers.asp).

But apparently there's a frickin' kernel patch needed to make it work.

Isn't a self-built NAS like this one (http://www.lundman.net/wiki/index.php?title=ZFS_RAID) a better solution?

Isn't a self-built NAS like this one (http://www.lundman.net/wiki/index.php?title=ZFS_RAID) a better solution?

I think he'd have to connect that with GbE, and he, understandably, wants to use his existing eSATA.

@QS: I meant just running solaris in a vm to handle the drive array so you can use ZFS. You'd still do everything else in your Debian host including accessing the iSCSI target.

Also, it might be worth the $$ to get a mini-SAS adapter. It would definitely give you more options. You can always get a single eSATA drive for OS and database backups.

If your not in hurry, you might want to consider trying to wait for USB3.0.

It should be just as fast as eSATA plus supports 4m cable lengths compared to eSATAs 1m(?) limit. Plus you can easily add more devices like you requested.

Unfortunately it's probably about a year away from having products even though there's already a kernel driver.

Port Multipliers (http://www.sata-io.org/technology/port_multipliers.asp).

But apparently there's a frickin' kernel patch needed to make it work.

SIL port multipliers have been supported in kernel since linux kernel .18 I have been using it since .16 but back then I had to roll my own kernel with patches applied.

Just get a port multiple type pci-e card that has the sil3132 or other SIL supported chipset and run the esata to an external case with 5in3 sata backplanes.

Hotswap worked for me last week when I had a drive die in the 5in3 enclosure.

Stuff I use that works:

*Port Multiplier [http://www.addonics.com/products/host_controller/ad5sapm-e.asp sata 2 port multiplier 1 to 5]
*Sata PCI-E card [http://www.siliconimage.com/products/product.aspx?id=32 PCI-e 2 channel esata card SIL3132 chipset]


my setup: http://mythtv.org/wiki/User:Blackoper

tux, NAS seems to me like a technology stacked onto a technology. It is a good thing for sure and has its place, but I'd like to avoid that. With your suggestion I was trying to think of a way to have one side of the mirror at the HTPC and the other remote, in case of fire or theft, but that doesn't seem to be possible with Linux MD RAID.

mm, again a VM & ZFS is a technology stacked onto a technology. And a mini SAS would mean my disks are different flavors, and I'd like to keep consistent.

Mac I hate USB. It's always been inferior to firewire, and still will be with 3.0. Its fundamental flaw is that it is master-slave, not peer-to-peer and daisy-chainable. It's like the difference between token-ring and ethernet; or Forrest Gump and Forrest Whittaker. I only use USB under duress.

Good to know blackoper that the addonics works; I see it everywhere. Also good to know that the kernel has it built-in now. But what I'm trying to find out is whether my mobo's e-SATA supports multipliers; I've got to think it has to, or else what is the use? Point-to-point one disk? Nah, I expect more from Asus.

No room in my case for a backplane.(above) I must reexamine the external cases I linked to above. What I need is one with a hot-swap backplane that will take SATA drives AND which I can run a SATA cable from each drive to the port-multiplier. That's the problem. I can't believe I have to jerryrig that.

I'm surprised to see that you would run a pre-rolled, 'tourist' OS like Mythbuntu? You know your stuff, so why? And what do you mean by 'Apache in secure mode'? And 'secure ftp'?

what's your motherboard type and sata chipset? I'm pretty familiar with all the ones that work with port multipliers. If you have open pci-e slots those two port SIL cards work great and have external E-sata connectors and internal sata connectors (and you can jumper which to use)


I'm surprised to see that you would run a pre-rolled, 'tourist' OS like Mythbuntu? You know your stuff, so why? And what do you mean by 'Apache in secure mode'? And 'secure ftp'?

Ease of use. Why install everything from scratch if you don't have to. Mythbuntu is pretty nice especially for it's diskless frontend booting features. I can combine the built in diskless server and my dd-wrt router for completely painless diskless frontends. I had the new zotac ion board fully working within minutes of turning it on. It's was almost plug n play except for adding shares to boxee and xbmc (and I'm sure I could make that automatic as well with some config file editing in both programs.)

secure ftp is: encrypted ftp server instead of the standard ftp that send usernames/passwords/files in clear text

secure mode apache is just https with everything passworded

I'm probably going to be moving over to linuxmce when the next version comes out that includes myth .22 I finally have all my home automation stuff ready to go. If linuxmce doesn't work well, though I'll be going the virtualization route and running multiple os's like mythbuntu, linuxmce, etc

Asus P5N7A-VM with external e-SATA port. Their support requires the board serial# just to ask a frickin' question, and that has delayed my asking it 5 days so far, just as they'd hoped.

You seem to invest alot less effort in your systems than I do.


And, yet another of my web hosters has collapsed. That makes 5 of my web hosting companies failed in 3 years. I've had no email for three days this time, before I figured it out. (been busy) I am sick and tired of paying them, and then having them disappear with my (admittedly measly) money. I am setting up my own hosting and email, in a Debian VirtualBox VM (for security) alongside my qb1ttorrent client. Don't know how many days it'll take to figure it out, but it is frickin' time.

No room in my case for a backplane.(above) I must reexamine the external cases I linked to above. What I need is one with a hot-swap backplane that will take SATA drives AND which I can run a SATA cable from each drive to the port-multiplier. That's the problem. I can't believe I have to jerryrig that.

Wouldn't the Addonics storage rack (http://www.addonics.com/products/raid_system/rack_overview.asp) do what you want?

Hadn't seen that, thanks. But this is an upscale home theater setup, and looking for something better than utilitarian... at least on the face. I guess I can paint the rest of it black if need be.

Where is the server/storage located at? Personally I have a problem with drive whine/fan loudness so I have them in a storage room that shared a wall with my home theater area. If you had another room to put the storage you could then probably save a lot of money by not worrying about the looks as much, plus you wouldn't hear the system

Wouldn't the Addonics storage rack (http://www.addonics.com/products/raid_system/rack_overview.asp) do what you want?
That looks awesome and quite versatile. Thanks, k_ross!

I put special care into making a completely silent system. I've recited my setup here several times, but nobody listens. My system is absolutely silent, at least to my old ears, and it belongs in the living room not least because the case display shows comprehensive system status.

Still thinking about remoting half the array. Will investigate further when I get time.

I guess you could replace the fans in those storage racks with quieter ones, or put a speed controller on the existing fans.

This is one amazing storage server (not really suited for the living room though):

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

67 terabyte 4U server for $7,867

Their choice of JFS for it is interesting, it confirms to me that JFS is the best choice for multi-TB filesystems.

Nice writeup on the Backblaze storage server, thanks tux.

I'm waiting patiently for the btrfs file system to be finished. Once it is, it should become the defacto standard for raid arrays.

About the backblaze storage server, I saw that earlier this week. Those 5 port backplanes are specially made by a chinese company and not available unless you order large amounts from what others who did the research were saying. Item: Chyang Fun Industry (CFI Group) CFI-B53PM 5 Port Backplane (SiI3726)

take that back here is an email about the backplanes:
Thanks your E-mail

CFI-B53PM backplane the price / quantity as follow:

50pcs US43

40pcs US43.5

30pcs US44

20pcs US44.5

10pcs US45

1pc US46

EMS freight depone the weight, so please let me know how many quantity you want, then I can evaluate the freight let you know, may 4~5 days arrive.

A high-quality (Debian) MD RAID FAQ:
http://git.debian.org/?p=pkg-mdadm/mdadm.git;a=blob_plain;f=debian/FAQ;hb=HEAD

OK, I am told by someone who seems to know (on the linux-raid listserv) that the reason my array is so darned busy when commercial-flagging is that the database is on the same array as the target video. (tho a different part)

He also says it is a fluke that I can boot to a RAID10 array, and that one drive would boot but the other won't, although I am skeptical about this because I've pulled the plug on each drive and I do get failover. But no one on the list contradicted him...

Nevertheless it seems that for performance (and ostensible boot reasons) I need a dedicated boot drive (probably a 2.5" WD Green), and the array handling /home. So my next decision was how to configure the array, and I've decided that it has to be a NAS with one side of the mirror in the HTPC and the other side remote in the garage. This way, if I ever have a theft or fire, I'll still have my library of videos and other backups. The garage unit will be down low somewhere, in case of fire everywhere) Apparently performance won't be a concern, as it will be gigabit ethernet, and the fastest the drive controllers can go is 3Gb/s; of course actual data acquisition from the drives is a small fraction of both. (far slower) I am told I can favor use of the local drive with the write-mostly option, so will investigate that.

Unfortunately with this choice I won't be able to get /home if a local array drive fails (network must be up before /home is mounted), but my goal is not 99.9999% uptime... it is data protection and performance.

So now I look for a remote case. Ideally it should be waterproof and dustproof, but I doubt I'll find that.

Waterproof and dustproof means airtight, which means you'll have a challenge keeping the components inside of it cool.

The Silverstone case has a large external passive heatsink, but it has other drawbacks. But this is what I have in mind if by some miracle it exists...

istarusa tells me that you must have a processor board inside the enclosure, as it is not possible to have a direct SATA to Gbethernet bridge; more complex than a SATA port-multiplier. This makes sense. However their fastest NAS is 100Mb, so they're out.

Oh, and aicipc and twinipc are the same company. twinipc seems to be the manufacturer.

Look at this little honey (http://qnap.com/pro_detail_feature.asp?p_id=110):
http://www.bsicomputer.com/new/nas/nas_images/ts-439pro/TS-439_Pro-intro.jpg
That's about what I need for the garage (no vents on top), and has 2x GbE (for network fail-over, load-balancing/bandwidth aggregation and multi- IP settings), 2x eSATA ports, and it'll take up to 4 IP surveillance cameras. About $750, but the processor is only 1.6GHz. There's a consumer-grade version (TS-419P, thermally-controlled fan, ~$590), but it's only 1.2GHz. Other companies offer enterprise NAS of 2.4GHz and 3.0GHz. But it is sure clear that qnap is competent.

That's about what I need for the garage (no vents on top), and has 2x GbE (for network fail-over, load-balancing/bandwidth aggregation and multi- IP settings), 2x eSATA ports, and it'll take up to 4 IP surveillance cameras. About $750, but the processor is only 1.6GHz. There's a consumer-grade version (TS-419P, thermally-controlled fan, ~$590), but it's only 1.2GHz. Other companies offer enterprise NAS of 2.4GHz and 3.0GHz. But it is sure clear that qnap is competent.

Those Qnaps are quite good, at least that's what I heard, haven't used them myself, but you could easily build something equivalent with a Atom mobo a small case and a 5-in-3 SATA cage (5 SATA drives arranged vertically in the space of 3 5.25 drive bays) like this one (just a random example, there are many similar ones available from many brands) for a lot less money:

http://www.pcconnection.com/IPA/Shop/Product/Detail.htm?sku=9227105

Major problem though, is NAS support. OK, so I find a good case, power supply, and cage, and install Linux and NX (remote desktop) on it.

I need to share those drives over GbE with minimal software intervention. Not about to use bloatware like NFS or Samba. sshfs or an SSH tunnel could not keep up the speeds I'd need. How do you suppose Qnap shares those drives securely? And provides all those other services? Sh*t is there a software update on their site? Eh yes. (downloading 69MB)

I may not have the time to invest in constructing a new system, although I'll look at their saveset to try and learn something.

Edit: Say, they do not necessarily say they share them securely... only that data's written on the drives in AES256. So how are they sharing them?

OK, the update comes down as a disk img file, which I can not mount in loopback. Probably encrypted, and knowing them in AES256, crafty bastards. I'm sure the password is a 17 digit string of random letters and numbers which would take 28 years to brute-force. I'd have to buy the damned thing to reverse it.

If you use a Via CPU/mobo combo you can get the power benefits of an Atom with a built in encryption accelerator that has full Linux support:

http://www.via.com.tw/en/initiatives/padlock/hardware.jsp

It looks like the accelerator puts out some pretty impressive numbers, although I'm not sure how they map with respect to SSH tunnel speed:

http://www.linuxjournal.com/article/8042

Can you run an SSH tunnel with AES?

How about iSCSI (SCSI over TCP/IP)? It makes the drives at the far end behave as local drives. I haven't tried it myself, but it might allow you to achieve your goal of a RAID 1 mirror with the mirrored drives physically remote.

How do you suppose Qnap shares those drives securely?

All these home/SME NAS devices like the Qnap use nothing else than Linux (or in some cases *BSD) with NFS and Samba.
I'm surprised you expect them to use something else!

But then what's wrong with Linux or *BSD for your NAS?
If you don't like NFS or Samba, use iSCSI (like k_ross suggests), I haven't used it yet so don't know much about it, but I believe it's faster than NFS/Samba.

drkdiggler, thanks for the info on Via. Via doesn't have a good rep for performance, so that'll take some investigation. I think I would tend first to an AMD or Intel reference design. Can definitely run a tunnel with AES; that's how I set up my sshfs shares & reverse tunnels for Myth, etc:
/usr/bin/ssh -v -f -l sleeper -i /home/sleeper/.ssh/id_rsa -2 -4 -c aes256-ctr,aes128-ctr -L 3306:localhost:3306 hex sleep 365d

k_ross thanks, iSCSI was suggested also by the gurus on the linux-raid listserv. No idea what it is, but I'll investigate. Also will see whether FUSE has a hot-sh*t solution.

tux, I wanted to avoid all software intervention in the NAS to cut out the delays, but it seems that is not possible if I expect to use GbE. Of course I'd run Debian in my NAS box if I do this. I should have expected a competent company like Qnap to use a current technology, not a 20th century solution like Samba or NFS. I'm getting a little queasy about going with a proprietary solution like Qnap, now that you mention it...

Oh, FUSE is amazing. Just look at these available modules:
Encrypted Filesystems (http://sourceforge.net/apps/mediawiki/fuse/index.php?title=EncryptedFileSystems)
CarvFs
Author: Dutch National Police Agency
Homepage: http://ocfa.sourceforge.net/libcarvpath/
Description:
LibCarvPath is a library for computer forensics carving tools. The library provides in the low level needs of zero-storage carving. It does this by providing an interface to hierarchically ordered fragment lists, and allowing these fragment lists to be converted to and from virtual file paths. These special virtual file paths can be used in conjunction with the CarvFS filesystem, a pseudo filesystem build using fuse and LibCarvPath.
MinorFs
Author: Rob J Meijer
Homepage: http://polacanthus.net/
Description:
MinorFs is a userspace filesystem (fuse) for Linux providing private storage to pseudo persistent processes. This allows programs that are run by a user to keep some data safe from all potential malware that runs with all this users privileges.
PhoneBook
Author: David McNab / david at rebirthing co nz
Homepage: http://www.freenet.org.nz/phonebook
Description:
PhoneBook is expressly designed for use in situations where someone can be under pressure (legal, military and/or criminal) to disclose decryption keys, and has a 'chaffing' scheme whereby the user can disclose only passphrases for non-sensitive material, and credibly deny the existence of anything else.

Media Filesystems (http://sourceforge.net/apps/mediawiki/fuse/index.php?title=MediaFileSystems)

Network Filesystems (http://sourceforge.net/apps/mediawiki/fuse/index.php?title=NetworkFileSystems)

And many more... (http://sourceforge.net/apps/mediawiki/fuse/index.php?title=FileSystems#Categories)

Barring some find in FUSE Networked Filesystems, it does look like iSCSI is the ticket as it can write directly to SCSI buffers using iSER (http://en.wikipedia.org/wiki/ISCSI_Extensions_for_RDMA). No hardware needed, just some kernel modules. And I can bring the network up on boot using initrd.img, so it's available for RAIDing.

Well I am simply unable to find a hot-swap case in the Qnap formfactor, much less one that is up to my quality standards. This stymies my efforts.

On a side note, I find that what I intend to build will technically be a SAN (iSCSI, fibrechannel, NBD, or AoE), not NAS (NFS, Samba, or AFS), since I will be using iSCSI. I've looked into NBD and eNBD, but they are older technologies than iSCSI. According to MySQL benchmarks, iSCSI outperforms even DAS SCSI!

Furthermore I find that there is a reliable Linux FUSE module for ZFS. So I will be using ZFS for my distributed video array, primarily for its checksumming, built-in RAID functionality, the ability to check the entire pool with a scrub (which is proportional to the size of the data I have), I can do backups by snapshotting, and other advanced features. I will create one pool in the HTPC with my four disks inside that case there. Then I'll export the four disks in the garage using iSCSI, and import these block devices into my HTPC. Then I'll add each remote disk as a mirror to the corresponding local disk in my ZFS pool.

I'd be using BTRFS if it hadn't betrayed me. Debian all-round, of course.

If I could just find an enclosure...

Did you have a look at the link I posted earlier in this thread:
http://www.lundman.net/wiki/index.php?title=ZFS_RAID

That enclosure seems perfect to me, if you want it small.

Otherwise, if it will be located in the garage anyway, any big tower case with loads of 5.25 slots (ideally 6 or 9) where you can fit 5-in-3 SATA cages should be fine, looks surely don't matter in the garage?

If you go ZFS then use Opensolaris, ZFS via FUSE is not ideal for a fast reliable media server, too many software layers, from what I heard it's fairly experimental and not widely used so probably not that well debugged.

But if you do iSCSI then the raid/fs will be all handled by your front-end not by the media server, so you would need to run Opensolaris on your front-end, so IMHO a media server running Opensolaris with ZFS and exporting the filesystems via NFS is the way to go. NFS can be very fast and is certainly reliable if set up correctly, Network Appliance (http://www.netapp.com/us/) servers have made a big business out of NFS servers, they are in use in many large corporations and are well regarded (I have worked with them), so NFS is not a limiting factor.
It might be old technology as you say, but Unix is even older and that doesn't make it bad! :)

I'd forgotten about that. Just found a great resource for enclosures here (http://www.servercase.com/Merchant2/merchant.mvc?Screen=CTGY&Category_Code=Mini-ITX). QBox-4 is $158, but without mobo and the PSU does not seem to be 80+. It says the bottom of the 4 bays is 50% taken up by the mobo, so it's really a 3 bay which is fine.

I don't want to use Solaris. The tail should not wag the dog. I'm still investigating the FUSE ZFS module, but so far it seems to be quite good, although development is slow. FUSE modules are very direct, and iSCSI is definitely the way to go. No need for NFS, Samba, etc, we're at the SCSI layer. I don't even need to partition the drives, just format them.

I'll need a boot device, and it's not worth a 2.5" drive, so maybe non-volatile SSD of some sort. USB boot? Is SSD fast enough these days?

My nodes are all named for constellations, so I guess the storage server would appropriately be called 'gemini'.

What do you have against solaris? Opensolaris+ZFS+COMSTAR=the ideal iSCSI target.

Also, I've read that some SSD's are outperforming HD's now, so one would be perfect for a boot drive.

I'm sure Solaris is very good, but I want a homogenous environment.

Well, that makes sense. I will say that if you use Nexenta, it's a 100% Debian environment, so the only thing different would be the kernel. And with that I'll drop it. :)

I'm looking real hard at that Backblaze setup for my app server prototype, but that PSU mounted on top might be in the way for a 2 or 4 CPU board. So I'm still considering storage options, and I'm glad you've got this thread going.

I looked at FreeNAS, NASLite, Openfiler, and NexentaStor, and don't like any for various reasons. Rejected NexentaStor because:
a. it costs, for more than 2TB, which I have;
2. it's OpenSolaris;
c. it's proprietary; and
4. problematic to add devices like security cameras (ZoneMinder (http://www.zoneminder.com/)) which I intend to add out there, although my Fund isn't selling due to this damned historic economic catastrophe which Republicans caused with their thieving policies. Notice there have been few consequences except job loss, with unemployment now at 9.8% (real rate estimated at 17%), so they're balancing this on the back of the Middle Class this time, which is tremendously unfair. But the Middle Class doesn't realize what's really happened, so why wouldn't they? As a result investors aren't investing, and my Fund isn't selling, so there's no buying.

Getting a little suspicious of the QBox-4, as it's not made anymore. And it's expensive for what it is. Still looking.

I find that ZFS-FUSE is rather CPU-intensive. Trying to find out whether it would overload an Atom 1.2 with commercial-flagging.

My questions and their answers on the ZFS-FUSE listserv by Emmanuel Anne:
What would using ZFS-FUSE mean from an operational standpoint? Once the FUSE module is in place would ZFS be just like another filesystem, with mkzfs, etc? Where do the tools come from? Would ZFS be available on boot, if I made say /home ZFS? Would I set up a regular boot drive, say ext3, then set up /home as raw disks with ZFS mirrored? Would performance on a 1.2GHz Atom & iSCSI be unacceptable for a video server?
A lot of these questions are answered on blog pages all around the web, but shortly :
- no mkfs.zfs it's handled by the zpool command
- the tools come from opensolaris of course
- not easy to boot on a root zfs partition, but to have /home or anything on zfs is easy, that's what I currently have here.
- performance on the cpu level is probably not an issue, memory is an issue though. You need at least 200 Mb of memory to be sacrificed to zfs-fuse (absolute minimum).

So would RAID1 be done through ZFS (as opposed to mdadm)? Is it proven?
Yes and yes (proven becuase the version on opensolaris has been around for many years, and sun is known for the stability of their code - for its complexity too !).

Is there a way to stripe -and- mirror, as with RAID10? Is it possible to RAID with only 2 drives, as it is with RAID10?
Yes, see the details on zpool man page.

I see that development of the ZFS FUSE module is rather slow. Does it have enough ZFS features? Compression? Checksumming?
Yes and yes.

Why is checksumming important when the drive does automatic bad block replacement?
There is a nice story on the sun site about how zfs allowed them to find a faulty power supply thanks to this.
And also this is the reason why ext3/ext4 force a fsck at regular intervals, here you won't need it.

I wouldn't recommend NexentaStor because it's a commercial distro. You can do the same thing for free and without limitations using Nexenta+ZFS+COMSTAR. Someone with your experience and resourcefulness should have no problem setting up an iSCSI target with those tools, and you wouldn't need to use ZFS-FUSE. If your cameras will be attached via USB and/or network, then you could run ZoneMinder in a headless Linux VM with Virtualbox quite easily. That way you'd have full access to the V4L/V4L2 stack. Virtualbox is surprisingly efficient, and now you can tell it to use any number of your CPU cores.

I know that I'm starting to sound like a Sun salesman, but this is really the best way to go for NAS. So I hope that you'll at least check the Opensolaris HCL before you buy your mobo so you can have the option to try it should you decide to do so.

I'll bite on your political rant by pointing the finger at both parties for not properly regulating Freddie & Fannie because they were (are) all getting kickbacks. I feel the same way about politics as I do about the Entertainment Industry: it should not be a "paying job". There should be no financial interest whatsoever. It should be something that you do AFTER you've retired from your *real* job and are financially secure.

Anyway, and staying OT haha, I've found a couple of reasonably obscure movies that I think you would also enjoy, QS: "Tetsuo the Iron Man (1989)" http://www.imdb.com/title/tt0096251/ and "Poultrygeist: Night of the Chicken Dead (2006)" http://www.imdb.com/title/tt0462485/

It is getting discouraging to find a quality case. I have one (maybe 3) choices, none up to my standards. This is getting into decision-deferral territory. I am surprised to find that it actually costs more to build a SAN system, than it does an actual media server. This is ********.


As to oversight, only one party was deregulating everything from financial services to airlines, to utilities, as fast as they could and wherever they could. This directly led to the catastrophe we have today. Reasonable regulation is what made this country strong, and only one party sees regulation as gumming up the works rather than keeping everybody honest. If someone doesn't see that, they are willfully blind.

More options:
http://www.chyangfun.com/pro04_1_1.asp
http://www.ioi.com.tw/products/proddetail.aspx?CatID=103&subcatID=5014&HostID=2027&ProdID=1030100http://babelfish.yahoo.com/translate_url?lp=ja_en&url=http://kakaku.com/item/K0000034857/
http://babelfish.yahoo.com/translate_url?lp=ja_en&url=http://kakaku.com/item/K0000034857/
(And something wrong with each...)

This is actually the "Qbox-4":
http://www.deca-tech.com/index2.php?pg01=525
http://shop.maxserve.co.jp/shopdetail/008000000003/order/
And the actual source:
http://www.chyangfun.com/pro02_1_8.asp

You know, I am coming to believe that just about any case with 4ea 5.25" bays and a power supply of at least 180w will accommodate a mini-ITX board and 3-to-5 drive cage. Question is, will the PSU have the right connexions to supply the mobo?

If this is the case then it broadens the solution set from just mini-ITX cases to all 4 bays.

I have found the following USB stick which seems PERFECT to me as a boot disk for a file server or a silent HTPC front-end, as it's SLC which means fast writes and 10x more write cycles than a normal MLC based USB stick:

http://www.disk2go.ch/ch_EN/pure2.php

It's quite affordable too, 45CHF (=45USD) for a 8GB SLC memory is good value, you will strugggle to find 8GB SLC CF cards at this price.

(ignore the encryption functionality which only works when used with some windows based software and is not really needed for a OS disk).

Nice, it's Swiss. And 8GB would be enough. I wonder whether the error correction is in the chips or the driver? And what brand chips they use? I'll ask.

I guess I'll have to plug it to a dedicated wire internally and just tie it up somewhere. This was what was nice about the old flashdisks, you plug it into IDE and it looked like it belonged. Also gotta make sure the mobo will boot to USB... my crappy old VIA wouldn't. I was wishing I could RAID1the boot drive, with the client HTPC for redundancy there... but the problem is they would both have the same personality. Same network name and settings, not to mention I customize each kernel to only that system's drivers. So that's hopeless.

I have had it with this DLink DIR-825 wifi router... network lockups every 1-2 days, with the 3 latest firmwares. Can't do with this in a SAN. I find they have been muggering up firmware lately. But gotta stay in 5.2GHz only for many reasons, so I am right now buying the new Netgear (http://www.netgear.com/Products/RoutersandGateways/RangeMaxWirelessNRoutersandGateways/WNDR3700.aspx). Had good luck with them, as they just license the pure reference design of chip manufacturers, and make that.

And I am on the verge of buying the Qbox4, as on close examination it seems fair quality. I've looked into adapting a non-ITX box, but internal space and backpanel accommodation are just too problematic. And there just isn't anything else acceptable.

Next step will be the 3-5 HDD cage. I think I'll stick with a name brand like SuperMicro, AEC, etc. And always cablesondemand SATA cables, of course. (Amphenol)

Then comes the dreaded mobo search. I'd like to get AMD this time, but if nothing comes close to Intel's stuff, OK.

....

I guess I'll have to plug it to a dedicated wire internally and just tie it up somewhere. This was what was nice about the old flashdisks, you plug it into IDE and it looked like it belonged. ....

If you have one of those sets of USB connectors that are designed to mount in an IDE slot then you could chop it up and reconfigure it so that the ports face into the case instead of out.

I woundn't buy one just to chop it up, I'd just use a cable like you mentioned. But if you have one laying around unused then it'd make a neater installation.

I did something similar to mount my MCE receiver within the window on my case.

Nice, it's Swiss. And 8GB would be enough. I wonder whether the error correction is in the chips or the driver? And what brand chips they use? I'll ask.

I'm sure the error correction is done in the chips, that's how it's done in any decent CF card that has ECC error correction, too.

I ordered one of those now, the 8GB version. I haven't decided yet what to use it for, but a 8GB SLC USB stick for the equivalent of less than fourty dollars (found it for 39CHF including shipping) was too good to resist. :D

I will do some speed tests etc. once I get it and let you know the results.

The only drawback for you could be, that it appears to be available only in Europe...

I asked my questions, and here's the ridiculous answer I got:
Support Request: CH-05545
Sprache: EN

Kundendaten: Mr Bill Southwell

Produkt: disk2go® PURE2 8 GB
Betriebssystem: Linux Debian Testing

Problembeschrieb (Type: A)
Is the error correction is in the chips, or the driver?

What brand chips do you use?
Hello

Please Check the disk2go® PURE2 Stick on a Windows PC

Let me know, if the stick works on the Windows PC correct or not.

best regards
Ihr disk2go® Support-Team
---------------------------------------------------------
Telefon: 0900 900 850 (CHF 2.50/Min.)
E-Mail : support@disk2go.com
I don't think he read my questions at all. I think he just came to the word "Linux", and got frightened...
And indeed it does look like an EU only product.

So I looked around and found Innodisk's SATADOM (http://www.innodisk.com/index.jsp), which looks like a great product although I'm afraid it's expensive. Also there's alot to be said for a USB stick, that can be used for boot or other purposes. So my next question is, is wear-levelling any more effective in a DOM than in a USB stick? A question for Innodisk to convince.


Edit: Whelp, Innodisk is going incognito... it is impossible to contact them. Filling out their webform and sending, results in:
We are sorry that this page is something wrong.
Other things to try:
* Search www.innodisk.com
Thaat's very helpful...
An email to support@innodisk.com gives:
<support@innodisk.com>:
220.130.35.236 does not like recipient.
Remote host said: 550 <support@innodisk.com>: Recipient address rejected: User unknown in virtual mailbox table
Giving up on 220.130.35.236.
And there is no phone number or address. I could track them down, but I suspect they want to be left alone...

I guess it's back into the arms of good old SanDisk.
(Save me SanDisk, saave me!)

I asked my questions, and here's the ridiculous answer I got:

The support is probably outsourced to India...
You shouldn't think that just because you contact a European company, you will get a response from a European... :rolleyes:

Also I'm sure the sticks are made in China like everything.
It might actually be available in the US, just under a different brand/name.

Anyway, I'll probably receive mine tomorrow, I'll let you know my impressions.


I guess it's back into the arms of good old SanDisk.
(Save me SanDisk, saave me!)

Trancend does very good flash products for less than Sandisk, I have many of their CF cards (one as OS disk), SSDs and USB sticks and never had any issue. None failed yet.

Yes I like and trust Transcend too. For some reason though, last time I bought a Sandisk Cruzer. I want to understand the wear level methods of each much better.

My HTPC has been down for a day, so no TV for me. Yesterday I installed a 2.5" WD Black 320GB, so I can lay down my / on a different device than the RAID10 array /home. For performance in Myth commercial-flagging you simply must have the database on a different device than the videos. I dutifully downloaded the latest Debian Testing and burned it to my last CD, but messed up the install at various stages until I had to go to work. Got home and re-re-re-discovered that the default desktop is Gnome, so now at 3:00am I have to go to the grocery store for some CDs so I can burn the KDE flavor and maybe get my HTPC running again....

Last night ordered the Netgear WNDR3700 and Qbox-4.

I'm sure the error correction is done in the chips, that's how it's done in any decent CF card that has ECC error correction, too.

I ordered one of those now, the 8GB version. I haven't decided yet what to use it for, but a 8GB SLC USB stick for the equivalent of less than fourty dollars (found it for 39CHF including shipping) was too good to resist. :D


Got the disk2go PURE2 8 GB now, I opened it up and found it uses an Alcor Micro AU6986AES controller chip: http://www.alcormicro.com/en_content/c_product/product_02b.php?CategoryID=3&IndexID=1

So hardware ECC error correction.

The flash chip is unbranded, it only says: RI-090826-64G
I guess the middle bit is the manufacturing date and 64G is 64Gbit=8Gbyte

hdparm -t rates the speed at 14MB/s (not bad for a USB stick, that's equivalent to a 96x CF card)

Alll in all it seems like a very good USB stick to use as OS disk for a small silent PC to me.

Edit to add pics of the USB stick internals:

Well done. I thought it might be Ricoh, but they don't make flash. Chances are it's an OEMed chip with proprietary number that we can't trace, although the controller says it's specifically for Micron "L52" SLC (http://www.micron.com/products/nand/mlc-slc) flash, which is the best brand there is for all types of memory. The flash is probably one of these (http://www.micron.com/products/nand/high_speed/partlist.aspx?den=8Gb) or these (http://www.micron.com/products/nand/partlist.aspx?den=8Gb).

I wonder whether it's running in USB2 mode? lsusb to get the assigned address, then
cat /sys/bus/usb/devices/usb{Device#}/speed
And try testing various readahead caches, and I bet you'll double that speed.

Got Debian configured last night on the HTPC's new Scorpio drive, and NX running so I can work on it remotely. (Read: 'in bed') Got /home pivoted over to the RAID10 array, and security set up. Rather than installing/deinstalling packaged Myth this time to set the system up (I always compile Myth), I just ran mc.sql then restored the database mythconverg backup so didn't lose anything.

I wonder whether it's running in USB2 mode? lsusb to get the assigned address, then
cat /sys/bus/usb/devices/usb{Device#}/speed

cat /sys/bus/usb/devices/usb{Device#}/speed gives me 480


And try testing various readahead caches, and I bet you'll double that speed.


What scripts/commands do you suggest?

Yep, it's running in USB2 mode if you checked the right device#.

Thought you saw when I'd posed this before, but for readahead buffer check the default:
# blockdev --getra /dev/sda
256
# blockdev --getra /dev/sdb
256

Then test it:
# time dd if={somelarge}.iso of=/dev/null bs=256k
32.3 MB/s
(if comes from the relevant disk)

Try some readahead sizes - 1024, 1536, 2048, 4096, 8192, and maybe 16384 (reboot each time, or disk cache will lie):
# blockdev --setra 4096 /dev/sda
# blockdev --setra 4096 /dev/sdb
# time dd if={somelarge}.iso of=/dev/null bs=4096k{etc}
77.7 MB/s

My HTPC:
1024 71.3 MB/s
2048 71.2 MB/s
4096 77.7 MB/s
8192 69.4 MB/s
16384 76.6 MB/s

When you get the best cache size, set it in rc.local .

[...]
Then test it:
# time dd if={somelarge}.iso of=/dev/null bs=256k
[...]


Thanks, the best I get is 14.8MB/s, which seems good to me.
The write speed is more interesting actually as it's SLC so should be much faster than most USB sticks. I will test that soon too.

OK, I guess you are reading at the max cell access speed. No positioning or latency time to eliminate.

SanDisk Extreme is rated at "up to" 30MB/s read and SanDisk Ultra is rated at 15MB/s read and 9MB/s write speeds. In practice, my SanDisk Cruzer Micro 4GB tests at 23.2 MB/s at the default readahead cache of 256KB. Not sure whether it's considered the Extreme or Ultra.

My freakin' RAID10 array, with 4096KB readahead on each sd drive and (an automatic) 8192KB on the md device tests at 1.6GB/s! Good news, and makes it worth it.

BAH! It tricked me. Even though I'd run the test before on the USB drive, and I'd renamed the file, it didn't fool the cache. When I rebooted I got the normal numbers, not no gigabyte. My best readahead cache size for the array with WD 2TB drives is 4096.

I also ran tests on the new Scorpio 2.5" 7200rpm boot drive:
512 84.6MB/s
1024 84.1
2048 83.9
4096 83.4
8192 83.4

Just as well because a smaller cache is better, as a boot drive will be dealing with small files, as opposed to the array which does videos. Heck, I'd better keep testing down.

I was going to say, looked a little too good to be true.

Perhaps I missed it in the thread, but why the choice of USB over a SATA or IDE to CF adapter?

Haven't selected USB yet, but is is more versatile than DOM. Didn't think of a SATA to CF adapter.

I was going to say, looked a little too good to be true.

Perhaps I missed it in the thread, but why the choice of USB over a SATA or IDE to CF adapter?

Simply because 40$ for a 8GB SLC flash device is cheap, a 8GB SLC CF card costs considerably more. The drawback is that this SLC USB stick is slower than typical SLC CF cards.

The Transcend Industrial CF card range is SLC and very well suited (I have a 8GB in my Linux home router/gateway), but quite a bit more expensive.

Ah sh*t, I've made a mistake. Whereas I was looking for an industrial-grade enclosure built fairly strong for rough environments, I just got this Qbox4 and it is very pretty. Lots of acrylic and very aesthetically appealing for gameboyz, it's not what I was looking for. Lots of crevices for dust, and fancy-pants goo-ga's. Well, I guess it's what I have.

Quality isn't terrible, it's just more -+*Style*+- than suits me. I guess I'll put a plastic bag over it to keep out the dust.

Ah sh*t, I've made a mistake. Whereas I was looking for an industrial-grade enclosure built fairly strong for rough environments, I just got this Qbox4 and it is very pretty. Lots of acrylic and very aesthetically appealing for gameboyz, it's not what I was looking for. Lots of crevices for dust, and fancy-pants goo-ga's. Well, I guess it's what I have.

Quality isn't terrible, it's just more -+*Style*+- than suits me. I guess I'll put a plastic bag over it to keep out the dust.

Sorry to hear that. you went through so many different ideas in this thread that it's hard to tell what you ultimately wanted.
I only suggested that case as a much less expensive do-it-yourself alternative to the Qnap NAS you were considering (which certainly isn't industrial grade and dust-proof either).
I have never tried it, but I read once that the trick to avoid dust is to make the case fans blow into the case instead of out of the case and then to add a filter in front of the case fans.
When the fans blow into the case, then it doesn't matter that the case is dust-proof as the air inside will try to get out again through the gaps, not be sucked in, so no dust comes in other than through the fans which have filters.

Not your fault tux, I simply ordered the best enclosure I could find. Decided you're right that a proprietary solution is no good. The need for an ITX box with open front hasn't been recognized yet, and this is just about the only choice.

This netgear wndr3700 router is built cheaply, even as it's about the most expensive there is. It'd better perform better than the dlink dir-825 without hangs...

tux, the suggestion about having all fans run as intake would probably be helpful. I was a semiconductor engineer in my previous life; the cleanrooms were run at higher pressure than the surrounding building to help prevent dust/dirt from entering. One thing to keep in mind is that your power supply probably blows out, so you may have to disassemble it to change the direction of the fan.

Yes I am suspicioning that the PSU fan will be the only one. I'll check its direction when I've decided on a mobo. The sides of the case are almost all vent, and there would be no plenum effect no matter where I put case fans.

As this severe filesystem corruption has happened to me twice now, with BTRFS and JFS, I am suspicioning the mobo. I have a very good power-conditioning transformer and battery backup on the system, so it wasn't a power glitch. It could be a mobo hardware failure, or it could be a design problem; but whatever, I am considering to replace it with a Gigabyte something-or-other.

No, as soon as I reinstalled jfsutils the array began working again, as if it was maybe missing a library for fsck.jfs or something.

Now that I believe BTRFS was not the reason for my prior loss of data, and that I am constructing a remote storage server, I think I'll try BTRFS again at least on one system. Checksumming is just so desirable, and I am concerned with the layering in ZFS-FUSE. Also, I'm not so sure about mdadm anymore as it didn't save me from this latest JFS filesystem corruption.

So I may use some function of BTRFS for syncing the two systems (HTPC & SAN), or maybe an rsync and checksumming function and let the SAN system sleep most of the time. This means that I'll need a mobo for it that does wake-on-LAN successfully and automatically, and this would pretty much eliminate the dust problem, being in the garage.

As this severe filesystem corruption has happened to me twice now, with BTRFS and JFS,
No, as soon as I reinstalled jfsutils the array began working again, as if it was maybe missing a library for fsck.jfs or something.


Ok I think I know what happened (not what caused the freeze but the superblock error you saw at boot afterwards).
Did the error you saw look like this?

fsck.ext3: Invalid argument while trying to open /dev/md0
/dev/md0:
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>

Close. It could not mount the JFS filesystem because it could not find a valid superblock on md2.

Close. It could not mount the JFS filesystem because it could not find a valid superblock on md2.

Ok forget the fact that my error message is for ext3, that's because my test filesystem was ext3 not jfs, but the cause is the same.

Basically that error happens when the underlying raid device hasn't been assembled yet. For various reasons it can happen that during boot when the OS is trying to check all filesystems that have a non-zero sixth field value in /etc/fstab (see man fstab) the raid isn't ready yet and therefore the necessary md0 (or in your case md2) device doesn't exist yet.

In other words, there was absolutely no filesystem corruption on your raid, the filesystem has never suffered any problems, only the raid was slow to reassemble (probably a consequence the freeze, ie unclean shutdown).
This is also supported by the fact that you didn't have to do any fsck to get the jfs filesystem mounted again.

Of course all this doesn't help finding the cause of your system freeze, but it should restore your confidence in md raid and JFS.

One thing that you might wanna investigate is why you missed the jfsutils package, that had probably been uninstalled by some debian update with dodgy dependencies (I seem to recall you are using debian-experimental?).

The array wasn't slow to assemble, as I'd left it in the half-booted state while I went to work. And yes maybe jfsutils had been deinstalled by an apt-get malfunction. It's the only plausible explanation.

But also understand what this episode implies: any filesystem glitch in the RAID10 volume destroys everything. I thought that building JFS on the array meant building it independently on the two mirror sides, but that doesn't seem to be the case. When I tried to mount the sd drives independently, mount didn't understand this RAIDvolume filesystem. And when I pulled the plug on each side respectively and rebooted I still could not see my data. True, in this case it was apparently a missing JFS lib, but what if the filesystem table got a glitch? No more nothin'. This was a shock to me. I thought I had two independent sides.

So I've lost confidence in mdadm, and am exploring other options. ATM I'm thinking the garage storage server can act as a backup, and I can rsync the HTPC array to that, with the --checksum option to confirm every file is truly backed up. When not in use the storage array could sleep. This would be an independent copy of all my hard-won data, in case of fire, fat-finger, or theft. I don't need the data instantly, as with RAID10.

You know, I was worried that my main HTPC would be overloaded, with Myth commercial flagging, transcoding, and storage server operations like ssh encryption and checksumming.

But clearly storage server operations should be carried out by the storage server. This'll be a little qbox-4 mini-ITX system in the garage in case of fire or theft. So I could iSCSI the HTPC's RAID0 volume to the storage server, hopefully through an SSH tunnel, and the storage server could merrily do the rsync, checksumming, and whatever operations, while the HTPC does its proper jobs. When not in use the storage server would sleep. Sure, the storage server will be just an Atom or Geode, but as long as its operations don't take more than 24 hours it's OK.

The array wasn't slow to assemble, as I'd left it in the half-booted state while I went to work.

I'm not sure what you mean by that, but I'm pretty sure that if you had done a cat /proc/mdstat right after getting the "invalid superblock" error you would have found that the raid was not assembled.


But also understand what this episode implies: any filesystem glitch in the RAID10 volume destroys everything. I thought that building JFS on the array meant building it independently on the two mirror sides, but that doesn't seem to be the case. When I tried to mount the sd drives independently, mount didn't understand this RAIDvolume filesystem.
RAID10 is not the same as RAID1 (mirror). With RAID1 you can mount each half of the mirror separately (I have done that in the past), with RAID10 you can't as the data on each half is striped.
You need to use RAID1 to have the kind of failure tollerance you want.


True, in this case it was apparently a missing JFS lib, but what if the filesystem table got a glitch?
It wasn't a missing JFS lib as the jfsutils package doesn't contain any libs, only utilities. You can mount a JFS filesystem without jfsutils installed, I just tried it here as long as it's clean. In your case it obviously wasn't clean as the box hadn't been shutdown cleanly so it required the jfs.fsck to reset it to clean.
Read up the following article to understand why JFS survives system crashes without any problem better than other filesystems:
http://www.ibm.com/developerworks/library/l-jfs.html

If the filesystem table (or rather superblock) got a glitch as you say, then you have a problem with any filesystem, but that doesn't happen easily as the superblock is stored in multiple copies.

{cough}
Moving on.

{cough}
Moving on.

I was just trying to make you (and anyone else reading this) understand better what happened, but if you're not interested then I'm moving on too...

So to jumpstart that moving on :-)

Have you given any considerations on whether you want to use wake-on-LAN vs. using the RTC to wakeup the machine? If so, I'd like to hear your thoughts. WOL is nice because the machine can wake up at any point if you want to run a backup for a certain reason, as opposed to waiting for a scheduled event. Also, how are going to deal with the time delay between sending the wakeup request and when all of the services are up and available? Simple 2 minute pause, or will you poll them until they are available?

Just today solidified that the storage server should perform all backup operations, not the HTPC, but haven't sorted out how yet. It should wake at a set time and do its work, accessing the HTPC's array through iSCSI. And it must not take more than 24 hours for a cycle.

I know Myth machines can go to sleep and use RTC for wake events, but I haven't studied that in detail as my machine is usually recording Dish. However I suspect something similar could be used for rsnapshot or similar. The storage server will wake itself and initiate its work, so timing after wake isn't a concern

I'm working on many other things (http://www.shroomery.org/forums/postlist.php/Cat/0/Board/1/page/0), but am striving for a best practice top-flight system here. I generally post progress here and welcome input.

And thanks tux, but that was a distressing episode that I'd just as soon avoid. I'm going to run memory tests, but expect to find nothing and will probably replace the mobo with a newer Gigabyte. Historically I haven't trusted Gigabyte and do Asus, but they are not allowed to corrupt my disks whether it's a hardware problem or not. OK maybe it was that jfsutils was deinstalled, but the result was still distressing.

I think I've identified all worthy candidates for mobo in my storage server to the below. The absolute minimum standard is at least 3 internal SATA ports and at least 1 GbE port.

I was hoping MSI had something good, but not one of their mobos has more than 2 SATA ports. This was a complete surprise and disappointment.

ASUS AT3N7A-I (http://www.asus.com/product.aspx?P_ID=xrR7wto9Z5BL42aU) (review (http://www.insidehw.com/Reviews/Motherboards/ASUS-AT3N7A-I/Page-2.html))
It's ION-based, which I don't really need because it will be headless, although it does come with CPU reducing that cost. But will that be enough of a benefit? Takes DDR2, which I prefer for now. $129.87

Gigabyte GA-2AIEV-RH (http://www.gigabyte.com.tw/Products/Networking/Products_Spec.aspx?ProductID=3054)
It's AMD which is desirable, and has the AM2+ socket which will take some of the newer AM3 CPUs. Has 6 SATA ports (!), and PCI-E 2.0 x16 rather than PCI as is usually the case. (and was dictated by Intel) Takes DDR2, which I prefer for now. Eh, Froogle, NewEgg, and eBay never heard of it, so can't pin down a price yet.

Gigabyte GA-6KIEH-RH (http://www.gigabyte.com.tw/Products/Networking/Products_Spec.aspx?ClassValue=Networking&ProductID=2788&ProductName=GA-6KIEH-RH)
This takes the Penryn /Merom processors, and although I don't know what wattage range these are, I'll bet they're low. Has 5 SATA ports (Silicon Image, very good), but PCI, which may not be a disadvantage as most surveillance camera cards are PCI, although it's a ghastly 32-Bit/33MHz. But look at this, it also has a mini card slot (PCI-E x1/ USB 2.0) and a mini PCI slot (PCI 33Mhz), for other uses like a wifi card. Also an IDE connector, 2 Serial connectors, 2 GbE (WOL, PXE, AFT, ALB , SLA, 802.3ad, SFT), and DDR2, but Intel IGP which I will hardly use. An astonishing $178 without CPU or boot device.

if you are looking into an atom board this one is pretty nice:
http://www.newegg.com/Product/Product.aspx?Item=N82E16813135247

pci-e 16x, pci-e 1x and a pci slot. Gigabit ethernet and 3 sata ports. The Atheros AR8131M lan chipset is supported in linux and from what I can tell freebsd 8 and opensolaris's/nexenta's newest version. If I get this I plan on throwing on freebsd 8 and using it as a zfs server. Power usage should be in the 30 watt range. With the pci-e slots and port multipliers you could run a lot of drives off this board (30+) I currently run about 12 drives and plan on expanding so the extra port multipliers are important to me.

I know it's an ecs, but the reviews and blogs seem to indicate the board is holding up well and not having any issues. The quality of their atom builds appears to be better than the normal stuff they do.

if you are looking into an atom board this one is pretty nice:
http://www.newegg.com/Product/Product.aspx?Item=N82E16813135247


That's a mATX board, quantumstate has a mini-ITX case to fill.

quantumstate of the 3 boards you mentioned the AMD board is probably best, it will consume more power than the other two, but Socket P processors are VERY expensive (unless you can find a second hand one on ebay for cheap) and the Atom+ION board doesn't have enough SATA ports.
Else you could get a 2 or 4 port PCI SATA controller card with the Atom board, but it might affect performance slightly because of the PCI bottleneck (might not be noticeable in practice).

edit: actually thinking of it, a 2 port PCI SATA controller won't be a bottleneck (133MB/s) and with the 3 on-board SATA connectors you have the 5 ports you need (you could use a IDE to CF adapter to boot from).
So an Atom board with 3 SATA on-board and a PCI slot with a 2 port PCI SATA adapter is probably the best solution for cost and power consumption.
The AMD board will likely idle at 70-80 Watts with a 45W TDP cpu and 5x 5400rpm 3.5" hard disks, while the Atom board will be around 50 Watts or less (always with 5 hard disks).

whoops, I thought I read over everything before I posted and just missed the itx part. A little destracted since I'm busy at work and multitasking with the forums stuff

Can't occupy the pci slot, as that's reserved for a camera board, however I could put on a port multiplier (adding to the cost).

Each of the boards has one problem or another I that don't like. The AMD board doesn't have a PATA port so I can connect a CDROM to load OS, and it doesn't seem to be available as no one has a price.

Are you going to be running the OS off of the CDROM all of the time or are you just installing once? Why not install from USB or via a network install?

Will be running the OS off of some sort of flash, so planning install media.

Have only one USB stick ATM, and that's being used. Assuming the BIOS can boot from the network, I didn't know a direct network install was possible. Only knew of the skeleton boot CD for network. Concerned for security with a direct net install; want shorewall up all the time. Besides, I'll bet any Debian Testing net install would be Gnome, not KDE.

Check it out (if memory serves correctly, you are a Debian guy):

http://wiki.debian.org/NetworkBooting

Will be running the OS off of some sort of flash, so planning install media.

Have only one USB stick ATM, and that's being used. ....

Black Friday is drawing near and you only need a 1GB stick to do the install. If you can hold out a couple of weeks then you should be able to pick up a 1GB stick for well under $10. Heck you might even have a friend with an 1GB stick lying around gathering dust.

Another option would be an external USB CD/DVD drive. It seems like that would be easier to remove than a PATA drive after the install is finished since it sounds like you won't want the unneeded install device sucking up power after the install is done.

I'll basically only need the install drive once, as Debian does rolling upgrades. I'm trying to avoid buying a slimline drive for the HTPC (which I could co-opt for the storage server) until the choices get a little better.

I can hold off for BF, but seems like only crap would be discounted like that. I'll try to look around, but time is limited. The AMD mobo hasn't been released yet. I hate Intel now, with all the crap coming out about their strong-arm tactics, in the anti-trust cases.

Will be running the OS off of some sort of flash, so planning install media.

Have only one USB stick ATM, and that's being used. Assuming the BIOS can boot from the network, I didn't know a direct network install was possible. Only knew of the skeleton boot CD for network. Concerned for security with a direct net install; want shorewall up all the time. Besides, I'll bet any Debian Testing net install would be Gnome, not KDE.

Just boot in install expert mode and don't select a window manager during the install stage. My light-weight manager for servers is "icewm". It runs quickly even on ATI RAGE 128 16Mb embedded cards. :)

But, the plus with buying Intel boards, is the NIC's that are on them. Worth the price savings alone.

Quantumstate, here is the thread you were looking for :)

Thanks. Anyone have any experience with the two mobos?
Jetway NC62K (http://www.jetwaycomputer.com/NC62K.html)
Zotac GF8200-C-E (http://www.zotacusa.com/zotac-gf8200-c-e-am2-am2-nvidia-geforce-8200-mini-itx-amd-motherboard.html)

Thanks. Anyone have any experience with the two mobos?
Jetway NC62K (http://www.jetwaycomputer.com/NC62K.html)
Zotac GF8200-C-E (http://www.zotacusa.com/zotac-gf8200-c-e-am2-am2-nvidia-geforce-8200-mini-itx-amd-motherboard.html)

don't have any expirience with those boards, but here is another one you could be interested in, integrated ATI Radeon HD 3200, but you mentioned that you don't care about the video:
JetWay JNC81-LF (http://www.newegg.ca/Product/Product.aspx?Item=N82E16813153124&cm_re=jetway_mini-itx-_-13-153-124-_-Product)

I have both of those boards. Two of the Jetway NC62K's, and one of the Zotac's. They both have different limitations.

I have both of those boards. Two of the Jetway NC62K's, and one of the Zotac's. They both have different limitations.

Care to elaborate?

Jetway NC62K

Cons:
- Single channel DDR2-800 memory support only, vs dual channel DDR2-1066 in the Zotac
- PCI instead of PCIe.
- Price

The single memory channel support was giving me problems with XBMC and an Athlon X2. I could only get smooth playback with the CPU frequency set at maximum. With mplayer and MythTV, I could set the CPU frequency to 1.8 GHz and get smooth playback. (With VDPAU of course). XBMC, it seems, put more stress on the memory bandwidth than mplayer or MythTV. Switching to the Zotac with dual memory channel allows me to run at 1.8 GHz and still get smooth playback in XBMC. So does using a Phenom instead of an Athlon X2, since the Phenom doesn't slow down the memory controller with the CPU frequency.

I also had a problem with USB in Windows on one of the Jetway boards, which I'm using for a car PC. I never bothered to figure out if it was a Windows driver problem, or a hardware problem, or even a power supply problem (possible). If you're interested in the details, see http://www.mp3car.com/vbulletin/review-palace/123163-jetway-nc62k-review.html

Zotac GF8200-C-E

Cons:
- Crappy analog audio quality. May not be relevant to everyone, if you're using digital audio out to a home theater A/V receiver.
- Crappy fan speed control. The first BIOS didn't have ANY fan speed control. A BIOS update allows you to set the fan speed to something like high, medium, or low. No way to adjust the fan speed based on temperature. Not through the BIOS, and not through software.
- No temperature sensors, at least none that lm-sensors (or Speedfan on Windows) can find.
- No heatsink on the voltage regulators. I added aftermarket heatsinks, so I could use my 95W TDP Phenom on it. :eek:

Overall, the Jetway is better quality. Temperature sensors, good fan speed control, heat sinks on the VRs, very good (but not excellent) analog audio quality. But single channel memory and PCI vs PCIe may be a limiting factor for some. As well as the higher price tag. And the USB issue is still outstanding, I never got around to narrowing down the cause of that problem.

Edit: added memory speed differences

Overall, the Jetway is better quality. Temperature sensors, good fan speed control, heat sinks on the VRs, very good (but not excellent) analog audio quality. But single channel memory and PCI vs PCIe may be a limiting factor for some. As well as the higher price tag. And the USB issue is still outstanding, I never got around to narrowing down the cause of that problem.

Edit: added memory speed differences

Thanks for the detailed report. Some of the problems you mention won't bother Quantumstate as he's not looking for a HTPC board, but for a file server mobo.

Personally I also think the Jetway is well suited for the use Quantumstate wants it for, the single channel memory should not be a problem for his purpose. Also I think he was explicitly looking to have a PCI slot, not PCIe.
The Jetway board also has the added advantage of the IDE connector which could be used for a boot disk, or better for a CF card to boot from, via a CF to IDE adapter.

OTOH I think the new Intel Pinetrail boards (might still have to wait a bit before the good ones with 4 SATA connectors come out) are best as the lower power consumption due to the northbridge integrated in the cpu (10-15 Watts in total), cannot be achieved by these AMD boards.

I'm waiting myself impatiently for the first 4 SATA port Pinetrail boards to come out, to build a very low power consumption NAS.

Yes thanks for the informative response k_ross. Looks like the Jetway is the way to go. Now, which CPU? Some Athlon 45W, I guess.

Not interested in any Intel products due to their abusive monopolistic behavior.

Edit: On second thought, some comments on Newegg for the NC62K really give me pause. Enabling both rj45's causes hangs and BSODs in Winduhs. And trying to use any USB port causes unstable operation, even with the latest firmware; it's not the driver, as a USB keyboard does not work reliably on boot. And it's said to run "HOT"; and there's criticism of build quality. These are issues I've learned to expect from non-Asus, non-Gigabyte mobos.

Reviews of the GF8200-C-E are much less negative, mainly revolving around DOAs and "blazing hot (85C at idle)" chipset.

Not many reviews on the NC81-LF and it seems like a viable choice, although lots of DOAs.