AVS Forum banner
1 - 20 of 43 Posts

·
Registered
Joined
·
1,005 Posts
Discussion Starter · #1 · (Edited)
I was thinking to myself, if Amazon Cloud Drive Everything Unlimited ($59/year) works as advertised, it would be an ideal, many TB, primary media storage solution. The solution should work for my situation and likely some others too. There are great benefits to storing many TBs of media in the cloud. Amazon is promoting Amazon Cloud Drive for exactly this use case.

I'm tickled to report that, indeed, I've been able to upload 17TB of media and everything is working as expected, as advertised -- so far.

If I was starting a multi-TB media storage solution today, I would certainly go with Amazon Cloud Drive instead of a NAS. I've probably spent $5,000 building up my NAS. The Amazon solution costs only $59/year.

The Problem
My movie server is a 24TB USB DAS. It's 17TB full of DVDs and Blu-Rays and still growing. The server is not RAID. Years ago I made a regrettable decision to implement a JBOD using Windows Storage Spaces "simple" mode. Once fully in, I discovered that "Simple" mode has no way to convert to RAID, retire a drive, or even recover from a single drive failure. Yuk! Thus I've thought of the movie server as a time-bomb waiting to blow. Much of the blame goes to Microsoft for making a service which is a few utilities short of robust.

The Painful Solution
The only way to defuse the time-bomb was inelegant and expensive. I had to rewrite the entire 17TB RAID. Even with falling disk prices and ever larger capacity drives, I was loath to spend more money on new hardware. I became resigned to the fact that at sometime the server would die and I'd be thrown back into viewing via streaming services, Netflix rentals, and a useless 24TB JBOD. I was loath to spend even more money on a movie server.

Enter the Cloud Solution
A new solution began to take form when Amazon offered "Unlimited Everything" cloud storage. For $59 a year, I could store all my DVDs, Blu-Rays, photos, videos, and backups in the cloud. For most, using this service would be a pain as it would take months to upload data. However, I had a fortunate situation. Here in France, Internet can be extremely fast and cheap. I pay $28/month for TV, telephone AND gigabit Internet. The actual throughput, measured by SpeedTest.Net, is often 920Mbps down and 210Mbps up. With those speeds, you can upload 2TB per day, or 17TB in eight days. This kind of bandwidth makes cloud based media storage practical. There's also MAJOR advantages over a local media server such as infinite storage, automatic backup, and anywhere access. Is Amazon's service worth the $59/year? You bet it is.

The Upload
I spent several days researching how to best upload 17TB of data. The limiting factor should be upload bandwidth so this should be the focus of optimization efforts. In actuality, you need to avoid other potential bottlenecks such as saturated CPU speed, I/O speed or limited temporary storage space. The solution, as so often happens with computing, was more complex than I expected. To upload that much data, every part of the system must be reliable and sufficiently fast.

Initially, upload speeds maxed out at 100Mbps even though the pipe could handle 200Mbps. I thought the shortfall might be an ISP or Amazon imposed limit. Turns out there were two other issues. First, Amazon's client software seems to limit upload speed to 100Mbps. The solution was to switch to CloudBerry Explorer Pro. Explorer was able to upload at the full 200Mbps. Secondly, my Intel (i3) NUC, for unknown reasons, was only sending data at 100Mbps. The solution was to switch to the inexpensive Gigabyte NUC (N2807 CPU, fanless), albeit having a much slower CPU, was able to upload at 200Mbps. The Gigabyte NUC, as it turns out, is sufficient to drive disk and Ethernet uploads at 200Mbps. The only issue is that I use NUCs headless, connecting via Splashtop Streamer, which slows uploads about 10%. The workaround was to stay mostly disconnected.

Once I had upload speeds nailed near the max, I found that Blu-Ray uploads were failing. The culprit is a poorly documented file size limitation of Amazon Cloud Drive. It appears that there's an upload size limit regardless of client software. The problem was solved by using software (RAR or 7-Zip) to split Blu-Rays into DVD size parts. I never investigated the actual size limit. I'm guessing it might be 5GB.

I had to be thoughtful about optimizing disk I/O. The movie server was reading data at 25MBps to 120MBps but mostly around 40MBps. I found the optimal solution, for my situation, was to have RAR read the ISOs from the server and write output to an SSD. Then have CloudBerry Explorer upload the split files from SSD. The system was thus able to sustain uploads at 200Mbps by reading from the SSD. Basically the SSD was used as a buffer. With this configuration, I've been able to sustain uploads at 200Mbps or 2TB/day. About 5% of the files generate some kind of upload error but CloudBerry software offers a nice retry feature.

I created several batch files to automated the uploading process.

1. A batch file which invokes RAR to split Blu-Rays into DVD sized files.
2. A batch file which contains a list of ISOs which call the above batch file.
3. A batch file which keeps my movie server drives from falling asleep. It works by changing the filesystems attribute flag every 280 seconds. This is necessary, for my situation, due to issues with my enclosures and Windows Storage Spaces.

Hardware Requirements:
1. Your DAS/NAS. Mine is a 24TB USB DAS JBOD driven by Windows Storage Spaces (not recommended).
2. Gigabyte NUC, N2807 CPU. I'm using Windows 10. This configuration has proven quite sufficient for 200Mbps uploading. Most computers with gigabit Ethernet and USB 3.0 will work fine too.
3. Optional, an SSD for buffering when DAS/NAS is too slow for splitting and uploading.

Services:
1. Fast Internet is a key to practical use. My ISP, here in France, is 1Gbps down, 200Mbps up for $28/month. For some, it might be a good idea to upgrade ISP service during the uploading period, perhaps just for one month.
2. Amazon Cloud Drive Unlimited Everything, $59/year.

Software:
1. CloudBerry Explorer Pro (recommended). It's one of the only clients that works with Amazon Cloud Drive. Nice feature set. Amazon's client is decidely less functional (not recommended at this time).
2. RAR software ($40) for splitting and joining files that exceed Amazon Cloud Drive file size limits (5GB?). Alternatives are Peazip (free), 7-Zip (free)

Upsides:
1. Unlimited storage.
2. Accessible everywhere - both uploading and downloading.
3. Always backed up.
4. Sell off part or all of your DAS/NAS.

Downsides:
1. Cloud copies of movies will take minutes (for me), perhaps hours, to download. This is not much of a problem if you can plan ahead.
2. Split Blu-Ray files, which are DVD sized, may take time to rejoin.
3. Amazon Cloud Drive terms of use essentially promises nothing. They reserve the right to not be held responsible for data loss, can discontinue service at any time, can limit service, and blah, blah.
4. ISP or Amazon may restrict bandwidth to something intolerable.
5. Amazon Cloud Drive may decide your usage of "Unlimited Everything" is intolerable to them. Of course this shouldn't be the case as that's what they advertise and what you're paying for. I'd feel better if there was more competition.
6. Uploading process may require some experimentation for optimal results. Probably will require periodic monitoring for errors and feeding of data.
 

·
Registered
Joined
·
23,130 Posts
Downsides:
1. Cloud copies of movies will take minutes (for me), perhaps hours, to download. This is not much of a problem if you can plan ahead.
2. Split Blu-Ray files, which are DVD sized, may take time to rejoin.
I'm sorry, but if it takes longer, and takes more work to play a rip off my "NAS" than it does for me to find the physical Blu-ray, stick it in the player, and wait through the forced trailers, that's a complete non-starter for me. After all, isn't that the whole point of having your Blu-ray/DVD collection ripped to a server? Instant access to any movie you own without having to wait for anything or do any work?
 

·
Registered
Joined
·
385 Posts
I'm sorry, but if it takes longer, and takes more work to play a rip off my "NAS" than it does for me to find the physical Blu-ray, stick it in the player, and wait through the forced trailers, that's a complete non-starter for me.
Wha? You mean they aren't streaming/directly playing from Amazon's storage, and instead I'd have to download a video every time I wanted to watch it? Fuhgeddaboudit.
 

·
Registered
Joined
·
680 Posts
Look into StableBit CloudDrive https://stablebit.com/CloudDrive
I have not used it, but this option does intrigue me too. bandwidth and speed limitations would be an issue for me though.
Cheaper than replacing hard drives. (not since the edit.)

edit: oops. Amazon CloudDrive in Canada is $500/year for 1 TB. They dont offer unlimited everything. USA does.
 

·
Registered
Joined
·
4,641 Posts
I'm currently testing the Stablebit CloudDrive with Amazon S3 storage, for encrypted, "important" stuff.

It's intriguing to say the least.
 

·
Registered
Joined
·
1,005 Posts
Discussion Starter · #6 ·
I just looked into Stablebit CloudDrive. It looks promising. However, they have a major issue with Amazon Cloud Drive. If and when they fix this issue, I'll happily take it out for a spin.

http://community.covecube.com/index.php?/topic/1340-amazon-cloud-drive-storage-chunk-size/

"We apologize for the issue.

This is mostly our fault.

Specifically, we need to get "production validation" so we can get off of just developer authorization. Otherwise, we're faced with a rate limit for upload and download speeds, and amount of traffic that is permitted. Since the documentation didn't really make this clear.... We didn't do this until after the product was released publicly. Additionally, currently the portal for getting the authorization/validation is apparently down. We're doing what we can to fix this, so that the provider is stable. But in the meanwhile, we'd recommend against using Amazon Cloud Drive as a provider."
 
  • Like
Reactions: Aryn Ravenlocke

·
Registered
Joined
·
761 Posts
This would be an interesting solution for an actual back-up of my media server. Currenly emby allows sycing to One Drive and Google Drive, but I cannot actually point my Emby Server at a cloud and use that as the primary source. Until that happens, I'm fairly certain I would still have a big machine in my walk-in closet so that I could serve media files up from it. However, if this works the way it is advertised, I would probably have forgone the cost of going with multiple parity disks and just relied on cloud storage as my off-site backup.

Had I been adding to the cloud as I ripped to the server, it might have been a practical solution for me. Alas, I only get around 20 MBps upload speed. That just isn't going to sut it for a collection over 40 TB in size. Also, it seems like any movie added that is over 20 GB won't take, even using the desktop loader. That eliminates hundreds of blu-rays.
 

·
Registered
Joined
·
1,879 Posts
To help with the redundancy with your server, you should still have back up of the "server" 17TB will fit nicely on 3x 6 TB USB external drives while you reconfigure storage with some redundancy.

RAID does not guarantee against data loss, it reduces the chance, since you have to have more drives fail that the fault tolerance of the array. You should still have back ups anyway. So, why not now?

Online storage is nice to have, but you have to have the connection to support it, which will cost more in the on-going basis than having local storage and back up.

500/500 (as long as Amazon cloud can support those upload/download speeds) plan from Verizon FiOS is $275/month. That buys you a lot of hard drives.
 

·
Registered
Joined
·
1,005 Posts
Discussion Starter · #9 ·
Correct that Amazon Cloud Drive has some size limit for uploading. The solution, as described in the OP, is to split the file into smaller parts.

Here's my progression of thinking. At first, I thought I'd use Amazon Cloud Drive to backup, then reload as a proper RAID. Then I thought why bother with a RAID, just leave the NAS as a JBOD as I don't really need to worry about disk failure. Then I thought why not just ebay the NAS keeping just a local 1TB for caching of my favorite media. I think a lot of people will walk through the same thinking.

I've now reached the conclusion that cloud drives are going to kill the home NAS.
 

·
Registered
Joined
·
23,130 Posts
Here's my progression of thinking. At first, I thought I'd use Amazon Cloud Drive to backup, then reload as a proper RAID. Then I thought why bother with a RAID, just leave the NAS as a JBOD as I don't really need to worry about disk failure. Then I thought why not just ebay the NAS keeping just a local 1TB for caching of my favorite media. I think a lot of people will walk through the same thinking.
But (I guess I didn't actually ask this before), what's the point of this if you have to effectively "rerip" the movie (download/reassemble) to watch it? The whole point of ripping Blu-rays is to have them available more quickly, and more conveniently than grabbing the disc off the shelf and stuffing it in the BD player.

I've now reached the conclusion that cloud drives are going to kill the home NAS.
It surely will, but not until the file size limitations are gone, and internet speeds/caps become a non-issue. I probably could do this, I've got 150/75 (down/up) so I could stream a Blu-ray if Amazon could keep up, but there's no point if I have to download and reassemble first, it would be faster and easier to just play the disk.
 

·
Registered
Joined
·
761 Posts
Correct that Amazon Cloud Drive has some size limit for uploading. The solution, as described in the OP, is to split the file into smaller parts.
Spending the time to go through and sort out hundreds of BD rips and then compress them (or breaking them into 3-5 parts) before uploading them, makes it an impractical solution for already existing systems. If the process is being done during an initial build of the media server, it might not be so bad/daunting.

I've now reached the conclusion that cloud drives are going to kill the home NAS.

Until file size limits are raised to at least 50 GB, and the ISPs stop placing such low limits on monthly data usage, a cloud solution for viewable media isn't going to be very practical. The solution already exists in a few different iterations for music, but storing multiple TB of movies in the cloud as a useful solution beyond non-streaming cold storage is yet some time off. Even using the cloud for strictly backing up a multi-TB NAS will not be terribly practical unless the allowed file sizes are increased though.
 

·
Registered
Joined
·
1,005 Posts
Discussion Starter · #12 · (Edited)
Will be interesting to see if StableBit Cloud Drive can live up to its claims specifically with Amazon Cloud Drive. If so, it will solve the issue of needing to split Blu-Rays, and will enable HD streaming. I really doubt they will get anywhere close to stutter-free Blu-Ray ISO streaming but maybe they can make work MKV or some other container.

With Amazon Cloud Drive Everything Unlimited being such a compelling media storage service, I predict we'll see many software vendors offering compatible solutions. Right now we have virtual drives (StableBit, odrive), sync (GoodSync), up/down loaders (Amazon, CloudBerry Explorer, Arq, Genie). I'm guessing there will be dozens more in six months. Surely we'll see cloud based Linux filesystems, similar to StableBit Cloud Drive. Also cloud enabled forks of mhddfs, aufs, and btrfs.
 

·
Registered
Joined
·
4,641 Posts
Well, while the "Cloud" storage IS an option, it's going to be tricky to use it as the end point. What I mean by that is, let's assume all of the logistical stuff is worked out, and you can store all your media on the cloud. Now, you have your tiny lil HTPC that's going to consume that media. So, you fire up your favorite front end and click on "movies". And then you wait.... :)

While Stablebit's solution is interesting that it caches the "most frequently accessed" data locally, it may not work as expected from a "media consumption" perspective. The "list" of your media is used very frequently and gets changes as well, and I'm not sure how Stablebit would handle it. We'll see.

I've actually been mucking around a fair bit with some custom software that does "media metadata caching" locally. What it does is, it maintains all the xml files (metadata), banners, backgrounds etc locally on each HTPC (I realize Kodi/Emby etc do something similar, but this is at a lower, file level), but maintains symlinks to the actual media file (which point to the actual location). This cache is pretty small, even for very large libraries, we're talking MBs not GBs and can be handled easily on HTPCs with minimal storage (i.e. small SSDs).

It'll be interesting to adapt it to Stablebit's model, where the media symlinks point to a location on the Stablebit CloudDrive, which in turn points to a cloud storage location. That way, you hit the cloud storage location only when you play the file, not browsing through your library.

Interesting stuff. :)
 

·
Registered
Joined
·
392 Posts
5mbit upload for 100$
No way I'm paying that much just so I can upload my collection and tie up my connection for like half a year.
 

·
Premium Member
Joined
·
5,818 Posts
I use Microsoft's One Drive for all my personal media. Since I'm an Office365 subscriber (for latest Office apps, OneNote, and Email accounts), they offer me unlimited storage for free. I'm sure Amazon's product is comparable.
 

·
Registered
Joined
·
385 Posts
Well, while the "Cloud" storage IS an option, it's going to be tricky to use it as the end point. What I mean by that is, let's assume all of the logistical stuff is worked out, and you can store all your media on the cloud. Now, you have your tiny lil HTPC that's going to consume that media. So, you fire up your favorite front end and click on "movies". And then you wait.... :)
Properly implemented, you'll wait about as long as what you would for Netflix, Amazon, et al make you wait when you bring up movies, tv episodes, etc. Those could easily be stored locally, or at least cached.

I think the two biggest hurdles once the data is on the cloud storage is can it stream while it's downloading, and can it keep up with very high bitrate video. The streaming services have this handled with adaptive quality, dropping to lower bitrate video if performance won't support higher quality. But you don't have that option when it's just a cloud drive storing the file.
 

·
Registered
Joined
·
981 Posts
I'll keep everything local thanks. Too many risks/problems with keeping all that media in the cloud. Cost per TB on drives is still falling, and I don't think I could handle converting all my full HD blu-ray rips to 720p or lower just to get the bit rate down to avoid buffering. Not everyone has 100Mb synchronous Internet pipes, Just the uploading would be so painfully slow in the UK as most consumer broadband installs have very limited upload speeds. I'm on fibre, and still only get 60Down/20Up.
 

·
Registered
Joined
·
1,390 Posts
Here in France, Internet can be extremely fast and cheap. I pay $28/month for TV, telephone AND gigabit Internet. The actual throughput, measured by SpeedTest.Net, is often 920Mbps down and 210Mbps up.
Internet here in the US is about 10 years behind Western Europe. There are a handful of towns that have gigabit residential service, but the majority of Americans don't even have more than one option for 10+ Mbps. The large companies refuse to compete with each other. It sucks.
 

·
Registered
Joined
·
1,879 Posts
Internet here in the US is about 10 years behind Western Europe. There are a handful of towns that have gigabit residential service, but the majority of Americans don't even have more than one option for 10+ Mbps. The large companies refuse to compete with each other. It sucks.
We also have a lot more distance between major population centers. France is smaller than Texas. And are we certain that the above speeds at the above price are available everywhere in France?


Google fiber is available in Kansas City, Provo and Austin, with gigabit service for $70/month.

We have Verizon FiOS and pay $59.99/month for triple play (Prime HD TV, Telephone, and 25/25 internet)
 

·
Registered
Joined
·
1,294 Posts
The state of the internet here is deplorable, unless you are lucky enough to have google fiber or live in a town with municipal fiber. Unbundling or heavy regulation to force competition is probably the only fix here, but will never happen with the lobbyist dollars these guys have. ATT and Verizon are letting their DSL customers rot, so they can push them to capped LTE with gross overage charges. I would guess Comcast will have their 300GB cap in place nationwide within a few years if not sooner. With these companies basically having a monopoly/duopoly there's not much you can do.
 
1 - 20 of 43 Posts
Top