AVS Forum banner

Please help! All sorts of eww...

945 views 8 replies 6 participants last post by  Aryn Ravenlocke 
#1 ·
I've searched the forums and found a number of topics that "sort of" touch on one issue or another, but nothing that is clear-cut pointing me in the proper direction. Given that I am in this current mess because there was no long-term vision in place to begin with, I think I am better off simply starting fresh(ish) and getting input from the beginning.


Although I don't have a firm number, I have what is likely 30 TB worth of movies and television on DVD and Bluray (mostly DVD). Once upon a time, my brother and I thought it would be cool to have a wall (then later two walls) that were completely covered end-to-end with discs organized in alphabetical order. Well, we ran out of space, and numerous titles have since been added to the collection. On top of that, the family has grown and we need more space. So, we began moving titles onto hard drives, figuring anything on the hard drive could then be put into a bin/tote and stored. Knowing very little about the process we went with what seemed easiest. Well, now we have 9 external hard drives that contain less than 20% of the collection. Furthermore, although the drives are hooked up to a PC on the home network, streaming from them is cumbersome, and only works in certain instances. Furthermore, we currently have no backup of the data that has been saved to the hard drives. If a drive fails, all the titles must be dug out and re-burned to a new drive, a process that will take days to accomplish.


We actually have modern equipment that is supposedly capable of accessing the library, but the problem is the way it is stored and how it is hooked up to the network. So, before things get any further out of hand, here's what I am looking to do, and I could really use some advice on how to make it happen.


First of all, I would like to stop using copious amounts of external hard drives. At this point, with the energy and space consumption, I am better off with a dedicated server. I'm okay with that. But I'm still running into problems because of how the files are currently saved, and also because I'm not entirely certain how best to proceed since I am trying to make a mostly clean start instead of continuing to Frankenstein a home streaming network together.


Here's what I currently have:



In the living room:
  • Samsung PN60E550D1f 60" 3D plasma Smart TV
  • Samsung BD-E5900 3D Bluray player
  • Sony STR-DH 810 AVR that supports passive HDMI 1.4
  • Acer AspireRevo AR3700-U3002 Slim and Compact Desktop PC running Power DVD 10
  • HP laptop with HDMI out running Power DVD 13



all are hardwired to the home router



In my brother's bedroom:
  • X-Box 360
  • PS3
  • both are hooked up to a Panasonic internet ready plasma



all are wirelessly hooked up to the home network



In my bedroom:


High(ish) power desktop PC that runs the copy software (AnyDVD)and that all the external hard drives are hooked up to. It is running Power DVD 10



What I want to be able to do is to stream content to the living room television and my brother's bedroom from one single unit/server. In order to improve quality, we intend to hard wire the television in his room to the network, but that is a final step as it will require going up into the crawlspace and it's a bit hot up there right now.


I don't have $26K to drop on a Kaleidescape server or I wouldn't be here, but I can afford to drop about $5-6K into the project.


Problems I am encountering:


No back-up

An increasing number of external hard drives that makes finding media more difficult

If the hard drive is not connected to a computer that is in turn connected into the television, the external drives are not recognized by the slim PC or the software on the television (it does have Samsung's All Share and also DLNA/UPnP identification).

The PS3 can sometimes find the drives, but the files are not organized by movie, rather every file on the drive is numerically ordered on the interface - making connecting through the PS3 useless.



***Currently, all DVDs are saved to the hard drive as Video_TS files

***Currently all Blurays are saved to the hard drives as ISOs


Like I said, we were going for simple and direct. Those are the two easiest ways to put the discs onto hard drives. Unfortunately, finding any sort of media playback that handles these (especially from a remote source) has been problematic. PLEX is an app available on the Smart TV, but it is does not recognize those files, and it has had some trouble identifying the external drives on the PC in the other room.


So obviously, the first thing I need is a new server or NAS. Asking four different people about the issue, I have received four different answers. Doing a bit of research, I'm leaning towards an UnRAID machine with all 10 drive bays filled with 4 TB discs, giving me 30 TB of internal storage after parity and cache are factored in. But will that actually do what I need it to? Or am I better off simply building a stupidly-large HTPC?


I guess I simply need some sound direction from some folks that have already figured things out. Do I need (or am I simply best-served) to invest in a media server, with all the processing horsepower to transcode on the fly? Or is there a NAS solution that will satisfy?


Do I need to go through and re-save every disc in a new format in order to facilitate a better, easier experience?


While I would prefer to stream from the television's built-in apps, I am not entirely against switching over to HDMI 3 and playing a movie or show from a low-power computer that is dedicated solely to streaming from the home media library. The problem I have with that solution is the lack of a strong interface solution. Sorting through a few thousand folder icons is tedious. I've seen the UI on programs like XBMC and PLEX, and would much prefer that sort of experience over the standard Windows Explorer format.


If I do need to change the saved file formats, is it possible to simply convert what is already saved without resorting to the physical disc again?


I need the Panasonic internet ready plasma to be able to stream the movies as well, either on its own, or through either the PS3 or XBox 360. Does this create further issues?




I'm sure there must be some kind of "relatively easy" solution. However, I don't know what it is, and the more I cobble solutions together, the worse I am making things.


Any and all help would be greatly appreciated.



-Hopeless in the Desert
 
See less See more
#2 ·
I'm not sure I can make any concrete suggestions - 30Tb is a metric **** ton of storage (using 16 3Tb disks and ZFS raidz3 (48 Tb raw, 35.5 Tb usable) ) . XBMC is very, very good but you need to use a media PC to run it ... I ran XBMC on an apple tv for a while but got tired of all the hackery it required. below that (arguably similar) are the linux streamers, like the Dune etc. The Dune will mount SMB file servers and display a plain directory view (which can get tedious if you have as many movies as you seem to). To get a neato "wall-o-film" you need to scan and import your files into a library management system like Zappiti or 10Muse ... this can be problematic and a pain in the ass the first few times you try it. I don't know offhand if Zapp supports TS files. I know it works with ISO Blu ray files. Below the media streamers are the Roku3 type devices, which are cheaper but not as powerful.


When my ReadyNAS filer broke, I used a Roku2 XS and Plex server and it was pretty good. I prefer the Zapp. interface but in a pinch, it served its purpose.


My recommendation would be: research and research some more a ZFS-based home file server (Freenas seems good), running in a decently large rack-mounted case like a Supermicro chassis and 16-24 3Tb drives, running ZFS Raidz3 . Supermicro server chassis come in large as 24 disk slots. Their fans are said to be quite loud, like a jet engine taking off! Don't just load linux on a server pc and create plain (non ZFS-raid ) partitions, disks fail and the movies lost will have to be re-ripped...


Try out XBMC with a Mac or PC or old Apple TV 2 (I guess) and see if you like it. Borrow a Dune from a friend nearby and try to play some of the TS files and ISO ... research 10Muse and Zappiti, they really are pretty good. Getting a file server built and consolidating the external drives into a single file server will help you a great deal. Backup can be a problem, unless you can light up some dark fiber and buy 2 of everything!


Blu Rays take up a tremendous amount of space. Perhaps depending on the size of your TV screens you could embark on using Handbrake and compressing them down ?
 
#3 ·

Quote:
Originally Posted by mjt5282  /t/1524436/please-help-all-sorts-of-eww#post_24534365


30Tb is a metric **** ton of storage... The Dune will mount SMB file servers and display a plain directory view (which can get tedious if you have as many movies as you seem to). To get a neato "wall-o-film" you need to scan and import your files into a library management system like Zappiti or 10Muse ... this can be problematic and a pain in the ass the first few times you try it.


Blu Rays take up a tremendous amount of space. Perhaps depending on the size of your TV screens you could embark on using Handbrake and compressing them down ?


It's been 2-3 years since we cataloged our library. Given what we had then and estimating the additions, we figure we are between 2500-2800 individual titles. But that's misleading given the amount of television and anime we have. Each season of a show is a title. So for instance, Stargate SG:1, X-Files, Buffy the Vampire Slayer, 24, and Friends comes out to 243 DVDs, each at a bit under 8 GB. SO the TBs add up in a hurry.



The size of the televisions are 60" and 50". I'm looking at the possibility of Remuxing the Blurays down so that they aren't taking up over 40 GB each, but that is a project for once the system is in place. Besides, it's going to take me some time to figure out what I am doing in that arena.


I have no issues with running a dedicated HTPC attached to the television if that is what it takes to get the best experience. I'm simply at the point that I want to get an idea of what the best solution for me is before I make things any worse for myself with constant experimentation. I'll look into XBMC and see what they are all about. I've also heard I should look at MyMovies. PLEX comes with an app on my television, so I had been considering that, although I get the impression that there are better ways to go about doing what I am trying.
 
#4 ·
You could crack open your external drives and use them as-is in a server running snapraid or flexraid. That will save you time re-ripping and give you some redundancy if a hard drive fails. From there you can add 3 or 4tb drives for more space.


A htpc would be the best solution for playback. The Intel NUC is tiny and silent but a bit pricey. Mediabrowser is a great front end.
 
#5 ·
[quote name="mjt5282" url="/t/1524436/please-help-all-sorts-of-eww#post_24534365"


Blu Rays take up a tremendous amount of space. Perhaps depending on the size of your TV screens you could embark on using Handbrake and compressing them down ?[/quote]


Ach! no. I'm firmly in the group that does not tolerate compressing content any more than it has been. Compress it now and in a few years when you go to display it on a 4K display you get a mess.


Anyway back to the OP's question, I think you need to invest in a few good media streamers that can handle the formats your content exists in now. And choose one format for all future rips. If your media player does not need content to be transcoded it makes the NAS situation simpler. I've got about a 10th the content you have stored on one of my PCs, and if I get much more I will have to look into a NAS, but I'm trying to avoid that cost for the moment. So my suggestions - 1. Choose a good media streamer 2. Hard wire 3. build a massive NAS.
 
#6 ·
@Aryn Ravenlocke

There are many different ways you could go, but given your shear volume of content, your desire for a multi-user network solution and your willingness to spend some money and not insist on "as cheap as possible" I would propose the following direction for both performance and future expandability.

A Media-PC controlling a disk farm


A lot of people like to say you can build/buy a HTPC for $300 but I have never seen a $300 class PC that I would want to use as an intensive media server -- not enough power and I don't mind spending some money. I built a Shuttle XPC SH67H3 with an I3 processor and 12GB RAM and Win-7 Pro for ~$500 (they still sell this but I'm sure they have newer or alternative models for you to peruse). This is my Media-PC. The Shuttle is built like a tank -- this I3 box has plenty of power to be a 24x7 server running multiple server apps. It can hold two internal HDD and an optical drive. On my GigE network I get sustained bidirectional network transfers between it and my main Win-7 desktop of 110MBps. I do most of my ripping, two disks at a time, on the two drives in my main desktop and shoot it across to the Media-PC. If I have a large volume I can always rip a third disk on the Media-PC. Another aspect of this configuration is that once set up, my Media-PC has no keyboard, mouse or monitor. It runs TeamViewer remote access software and is accessed/controlled over the network from my main desktop -- for a while it was running in a corner of the basement until I made room for it in the main computer room upstairs.

Storage:

In my case I put a pair of 2TB 7200 rpm SATA III drives in the Shuttle along with an LG BD burner. The Shuttle has 4 x USB-3 ports (5 x USB-2) and a PCI-e slot for an eSATA card with a port multiplier. I put in a Mediasonic 8-drive 2-port eSATA III card and attached a Mediasonic 4-bay enclosure (with the future expandability option to add a second 4-bay to the other eSATA port on the card). The 4-bay enclosure is populated with 4 x 3TB 7200 rpm SATA III drives so my current capacity on that server is 16TB with easy expandability to 32TB with another 4-bay enclosure and 4 x 4TB drives. Actually, in my case I would use 3TB drives because I like 7200 rpm and 4TB drives are all 5900 rpm. For power savings, Win-7 is set to spin down any drives that are idle for 30 min and it does so quite well and selectively -- if I only access one of the sleeping drives in the enclosure, it only wakes up the drive I want to access.


In your case (for max storage) I would install a pair of 4TB drives in the Shuttle; skip the eSATA card and attach a Mediasonic 8-bay USB-3 enclosure ; populate it with 8 x 4TB drives = 40TB of storage. The future option would be to add an additional 8-bay USB-3 unit with an additional 8 x 4TB = 32TB. Do you think 72TB would fit the bill for your collection? In this entire scenario, the best part is that you don't have to completely populate the bays all at once -- you can add drives as needed to the second 8-bay unit -- I'm assuming that with your volume you will probably want to fully populate the first 8-bay unit at purchase. If you want to go beyond 72TB, you have another two USB-3 ports on the Shuttle.

Drive Aggregation and Protection vs. Content Backup:

Backup means a separate copy of the media. In the simplest sense, your optical disks are the backups for the Media-PC server. You probably don't want to entertain the thought of re-ripping 4TB of disks if one of them fail on you. You have two good choices:
  1. Duplicate HDD enclosures to mirror the primary HDD's
  2. A software RAID solution


A set of mirrored HDD's will give you protection from multiple primary disks failing simultaneously but the cost is huge. A software RAID solution will also give you protection from multiple disks failing -- if you configure it that way. Whereas the cost of mirrored HDD's is one drive for each drive to back up, the cost of software RAID is 1 parity drive for every simultaneous HDD failure you want to guard against.


For a Media-PC I think FlexRAID is the way to go. First off, FlexRAID will provide drive aggregation so you can bundle the identities of multiple drives into being represented as a single drive letter volume. This is not the same as when an OS creates an expanded volume by combining multiple drives at the hardware level with a single index. With FlexRAID, the drives maintain their individual identities and can still be assigned individual drive letters and accessed individually if desired. Second, the RAID portion will create a parity drive(s) to protect the array. Should a drive fail, you insert a replacement and the system rebuilds the array. Multiple parity drives protect against simultaneous drive failure. And of course in the event of catastrophic failure beyond the limits of your protection, you only lose the content on the drives that fail and you have the disks as your ultimate backup.


Hope this gives you a feel for one possible solution and shows you a direction that will serve you in the future as the collection grows.
 
#7 ·
I appreciate all the input, both here and through PMs. It looks like my mess isn't quite as bad as I thought it was, though it by no means will be a "simple" fix. The more I have asked around, the more I have been led to believe that unRAID is not the ideal solution for those of us that are not IT-inclined. So, here's what I am considering, hoping that it will work the way I imagine.

Storage:


Synology DS 1813+ with a DX513 expansion

or

Mediasonic 8-drive H82 ProBox (2 of them)


Hardware:

Server PC:


Shuttle XPC SH67H3

or

Intel NUC


Regardless of which I choose, I'll be going with an i3 processor to help give it enough oompf to handle the task.


I'm still going back and forth between a Linux box or a Windows 7 box for this part. An advantage of the Windows 7 box is that every other PC in the house is Windows 7 or later, so there should be ZERO compatibility issues.


HTPC:


Intel NUC

or

Zotac Z Box

or

Gigabyte (generic i5 system)


This would be connected to the television and hard wired to the media server.


Software:


MyMovies on the server paired with XBMC on the front end with the PLEX plug-in for those rare times I am away

or

Media Browser 3



I need way more info and sampling before I can decide between these. The available scrapers for either set-up seem to be woefully inept for television series, so I might have to keep looking. Otherwise I have a TON of manual entry to get to.



Am I missing anything? I figure I'll poke around the forums some more and probably start separate threads in the appropriate places to get help with deciding which option in each category.
 
#8 ·
If you haven't already, take a look at Yadis for your library front end. I had a heck of a time with Zappiti and 10Muse when it came to TV series, but have had really good success with Yadis. Some people don't care for it's lay out (single screen for series, seasons displayed as number, then scroll through season episodes) but I personally prefer it to a wall of episode icons, and again the scraping was the most successful of the 3 I've tried.
 
This is an older thread, you may not receive a response, and could be reviving an old thread. Please consider creating a new thread.
Top