Mfusick's How to build an affordable 30TB Flexraid media server: Information Requested.! - Page 144 - AVS | Home Theater Discussions And Reviews
Forum Jump: 
Reply
 
Thread Tools
post #4291 of 4316 Old 02-27-2015, 06:10 PM
AVS Special Member
 
Ken.F's Avatar
 
Join Date: Jul 2010
Location: West Rockhill, PA
Posts: 2,233
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 337 Post(s)
Liked: 346
Quote:
Originally Posted by ajhieb View Post
A little googling for that LocalSystemNetworkRestricted shows a few cases where SuperFetch was getting out of line. That may or may not tie into what your problem is.

Just out of curiosity you haven't done anything odd like shrink or remove your pagefile have you?
The virtual memory is set for 24072MB and is located on the 4TB HDD that isn't in the pool.

I tried disabling superfetch during a validate about a month ago. That didn't seem to make any difference either.
Ken.F is offline  
Sponsored Links
Advertisement
 
post #4292 of 4316 Old 02-27-2015, 06:15 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,835
Mentioned: 8 Post(s)
Tagged: 0 Thread(s)
Quoted: 571 Post(s)
Liked: 483
Let's change gears.

If you downloaded the sysinternals suite, go ahead and run RAMmap (I think it might need to be run as admin) then click on the processes tab, sort by the "Modified" column and see exactly what is allocating all of that memory and not using it.

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #4293 of 4316 Old 02-27-2015, 06:22 PM
AVS Special Member
 
Ken.F's Avatar
 
Join Date: Jul 2010
Location: West Rockhill, PA
Posts: 2,233
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 337 Post(s)
Liked: 346
I don't see anything unusual there.
Attached Thumbnails
Click image for larger version

Name:	RAMmap.jpg
Views:	14
Size:	448.4 KB
ID:	573577  
Ken.F is offline  
post #4294 of 4316 Old 02-27-2015, 06:27 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,835
Mentioned: 8 Post(s)
Tagged: 0 Thread(s)
Quoted: 571 Post(s)
Liked: 483
This is getting weird. I'm really at a loss here. I've never seen everything report such wildly conflicting information.

Might be time to call in a priest.

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #4295 of 4316 Old 02-27-2015, 06:33 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,835
Mentioned: 8 Post(s)
Tagged: 0 Thread(s)
Quoted: 571 Post(s)
Liked: 483
Quote:
Originally Posted by Ken.F View Post
I don't see anything unusual there.
If you look on the "User Counts" tab in RAMMap does it account for the large chunk of modified memory there?

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #4296 of 4316 Old 02-27-2015, 06:37 PM
AVS Special Member
 
Ken.F's Avatar
 
Join Date: Jul 2010
Location: West Rockhill, PA
Posts: 2,233
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 337 Post(s)
Liked: 346
Quote:
Originally Posted by ajhieb View Post
This is getting weird. I'm really at a loss here. I've never seen everything report such wildly conflicting information.

Might be time to call in a priest.
Yeah, I think it needs a Flexorcism.
I have no idea what's eating the memory but it only happens when FlexRAID runs a validate or verify. It's 100% repeatable, and I can make the problem go away temporarily by pausing the operation for a while. Thanks for trying.
ajhieb likes this.
Ken.F is offline  
post #4297 of 4316 Old 02-27-2015, 06:40 PM
AVS Special Member
 
Ken.F's Avatar
 
Join Date: Jul 2010
Location: West Rockhill, PA
Posts: 2,233
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 337 Post(s)
Liked: 346
Quote:
Originally Posted by ajhieb View Post
If you look on the "User Counts" tab in RAMMap does it account for the large chunk of modified memory there?
Paged Pool is pretty high.
Attached Thumbnails
Click image for larger version

Name:	RAMmap 2.jpg
Views:	15
Size:	321.2 KB
ID:	573625  
Ken.F is offline  
post #4298 of 4316 Old 02-27-2015, 07:05 PM
AVS Special Member
 
Ken.F's Avatar
 
Join Date: Jul 2010
Location: West Rockhill, PA
Posts: 2,233
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 337 Post(s)
Liked: 346
Validate failed. I got a HANDLE_VALUE error=32 on a file that was in use by a remote Plex user. Nothing alarming about that. I have validates scheduled to run in the middle of the night so that usually doesn't happen.
Ken.F is offline  
post #4299 of 4316 Old 02-27-2015, 07:10 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,835
Mentioned: 8 Post(s)
Tagged: 0 Thread(s)
Quoted: 571 Post(s)
Liked: 483
This is a real head scratcher.

I can't find any other examples at all of people having that much Modified, Paged Pool memory being allocated. Several non-paged pool, but that seems to be a different issue.

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #4300 of 4316 Old 02-27-2015, 07:25 PM
AVS Special Member
 
Ken.F's Avatar
 
Join Date: Jul 2010
Location: West Rockhill, PA
Posts: 2,233
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 337 Post(s)
Liked: 346
Quote:
Originally Posted by ajhieb View Post
This is a real head scratcher.

I can't find any other examples at all of people having that much Modified, Paged Pool memory being allocated. Several non-paged pool, but that seems to be a different issue.
I might try deleting the configuration and start over.

I'd like to ditch cruise control and go advanced and run two pools but the FlexRAID wiki is incomplete on that topic.
Ken.F is offline  
post #4301 of 4316 Old 02-27-2015, 07:33 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,835
Mentioned: 8 Post(s)
Tagged: 0 Thread(s)
Quoted: 571 Post(s)
Liked: 483
Quote:
Originally Posted by Ken.F View Post
I might try deleting the configuration and start over.

I'd like to ditch cruise control and go advanced and run two pools but the FlexRAID wiki is incomplete on that topic.
From what I can tell on that topic, I don't think it's supported (as in: it's disabled) until the next release.

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #4302 of 4316 Old 02-27-2015, 07:42 PM
AVS Special Member
 
Ken.F's Avatar
 
Join Date: Jul 2010
Location: West Rockhill, PA
Posts: 2,233
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 337 Post(s)
Liked: 346
It let me get started setting it up advanced some time ago but I hit a snag. I don't remember what it was that tripped me up but remember trying to look it up on the wiki. It had a link to read detailed instructions about whatever it was but the link was broken. I ended up going back to cruise control.
Ken.F is offline  
post #4303 of 4316 Old 02-27-2015, 07:47 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,835
Mentioned: 8 Post(s)
Tagged: 0 Thread(s)
Quoted: 571 Post(s)
Liked: 483
Quote:
Originally Posted by Ken.F View Post
It let me get started setting it up advanced some time ago but I hit a snag. I don't remember what it was that tripped me up but remember trying to look it up on the wiki. It had a link to read detailed instructions about whatever it was but the link was broken. I ended up going back to cruise control.
Yeah, advanced works, but he's evidently disabled the ability to make more than one pool.

http://forum.flexraid.com/index.php?topic=280.0

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #4304 of 4316 Old 02-27-2015, 07:51 PM
AVS Special Member
 
Ken.F's Avatar
 
Join Date: Jul 2010
Location: West Rockhill, PA
Posts: 2,233
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 337 Post(s)
Liked: 346
Quote:
Originally Posted by ajhieb View Post
Yeah, advanced works, but he's evidently disabled the ability to make more than one pool.

http://forum.flexraid.com/index.php?topic=280.0
OK, that might have been the part that tripped me up.

Thanks again for your help.
Ken.F is offline  
post #4305 of 4316 Old 02-27-2015, 07:52 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,835
Mentioned: 8 Post(s)
Tagged: 0 Thread(s)
Quoted: 571 Post(s)
Liked: 483
Back to your memory problem,

This seams similar, but there is no resolution there.

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #4306 of 4316 Old 02-27-2015, 08:50 PM
AVS Special Member
 
Ken.F's Avatar
 
Join Date: Jul 2010
Location: West Rockhill, PA
Posts: 2,233
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 337 Post(s)
Liked: 346
Quote:
Originally Posted by ajhieb View Post
Back to your memory problem,

This seams similar, but there is no resolution there.
That looks very similar. I'm not familiar with MmSt tag so I did a little looking around and found this:
http://blogs.technet.com/b/askperf/a...-pool-tag.aspx

It looks like it might be helpful. I'll have to look at it closer tomorrow.
Ken.F is offline  
post #4307 of 4316 Old Yesterday, 12:54 AM
AVS Special Member
 
DotJun's Avatar
 
Join Date: Dec 2012
Posts: 1,121
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 237 Post(s)
Liked: 68
Quote:
Originally Posted by Ken.F View Post
Thanks, I'll look into it.

It bugs me a little but I'm not losing any sleep over it. I can get it to pass the validate and verify if I check on it every three or four hours.
My memory use is mostly "modified" (memory that must be written to disk before before it can be used for another purpose). If I pause the validate operation for 15-30 minutes the memory use goes down by about half and stays that way. Shortly after resuming the validate operation the memory use will quickly drop down to normal levels, then slowly start climbing again.
This is why I opted out of flexraid-f. Too many posts of people having validate/verify problems. I'm leaning towards snapraid for flexraid transparent raid now.

Quote:
Originally Posted by Mfusick View Post
I always thought Flexraid just uses a lot of memory. You sure something is wrong? What is your specs?
Isn't flexraid limited in memory usage due to being 32bit?

Quote:
Originally Posted by ajhieb View Post
A little googling for that LocalSystemNetworkRestricted shows a few cases where SuperFetch was getting out of line. That may or may not tie into what your problem is.



Just out of curiosity you haven't done anything odd like shrink or remove your pagefile have you?
You can disable superfetch and its sister to test out your theory.
DotJun is online now  
post #4308 of 4316 Old Yesterday, 06:03 AM
AVS Special Member
 
Ken.F's Avatar
 
Join Date: Jul 2010
Location: West Rockhill, PA
Posts: 2,233
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 337 Post(s)
Liked: 346
Quote:
Originally Posted by DotJun View Post
Isn't flexraid limited in memory usage due to being 32bit?
I still haven't figured it out but I doubt FlexRAID itself is using the memory. FlexRAID is reading many large files at the same time and for some unknown reason that is causing the system to run out of memory.

Quote:
You can disable superfetch and its sister to test out your theory.
I tried disabling superfetch but that didn't make a difference. Thanks for the ideas.
Ken.F is offline  
post #4309 of 4316 Old Yesterday, 09:35 PM
AVS Special Member
 
Dark_Slayer's Avatar
 
Join Date: May 2012
Posts: 2,929
Mentioned: 5 Post(s)
Tagged: 0 Thread(s)
Quoted: 381 Post(s)
Liked: 340
I have 16GB of ram in my server and Flexraid uses around 4 all the time. It's some kind of write cache, because when I write a file smaller than that amount of ram it's extremely fast
Dark_Slayer is offline  
post #4310 of 4316 Old Yesterday, 10:23 PM
AVS Special Member
 
ajhieb's Avatar
 
Join Date: Jul 2009
Posts: 1,835
Mentioned: 8 Post(s)
Tagged: 0 Thread(s)
Quoted: 571 Post(s)
Liked: 483
Quote:
Originally Posted by Dark_Slayer View Post
I have 16GB of ram in my server and Flexraid uses around 4 all the time. It's some kind of write cache, because when I write a file smaller than that amount of ram it's extremely fast
Did you switch to tRAID or is that for RAID-F?

RAID protection is only for failed drives. That's it. It's no replacement for a proper backup.
ajhieb is online now  
post #4311 of 4316 Old Yesterday, 10:53 PM
AVS Special Member
 
Dark_Slayer's Avatar
 
Join Date: May 2012
Posts: 2,929
Mentioned: 5 Post(s)
Tagged: 0 Thread(s)
Quoted: 381 Post(s)
Liked: 340
Quote:
Originally Posted by ajhieb View Post
Did you switch to tRAID or is that for RAID-F?
Still raid f at the moment. Switched back and forth. Going back to T again when I get time for a rebuild
Dark_Slayer is offline  
post #4312 of 4316 Old Today, 02:25 AM
AVS Special Member
 
DotJun's Avatar
 
Join Date: Dec 2012
Posts: 1,121
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 237 Post(s)
Liked: 68
Quote:
Originally Posted by Dark_Slayer View Post
I have 16GB of ram in my server and Flexraid uses around 4 all the time. It's some kind of write cache, because when I write a file smaller than that amount of ram it's extremely fast
That's right, flexraid should be limited to 4gb due to its 32bit nature.

Quote:
Originally Posted by Dark_Slayer View Post
Still raid f at the moment. Switched back and forth. Going back to T again when I get time for a rebuild
Can someone correct me here about the different iterations of flexraid? As far as I know, there is raid-f and raid-t which are both part of the snapshot raid product and then there is the more traditional realtime raid called transparent raid?
DotJun is online now  
post #4313 of 4316 Old Today, 06:17 AM
AVS Special Member
 
Ken.F's Avatar
 
Join Date: Jul 2010
Location: West Rockhill, PA
Posts: 2,233
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 337 Post(s)
Liked: 346
Quote:
Originally Posted by DotJun View Post
Can someone correct me here about the different iterations of flexraid? As far as I know, there is raid-f and raid-t which are both part of the snapshot raid product and then there is the more traditional realtime raid called transparent raid?
FlexRAID only has two versions. RAID-F is snapshot and tRAID is transparent (realtime). Neither FlexRAID product is traditional RAID.

See:
Table comparison of Transparent RAID vs RAID over File System
Dark_Slayer likes this.
Ken.F is offline  
post #4314 of 4316 Old Today, 06:39 AM
AVS Special Member
 
Ken.F's Avatar
 
Join Date: Jul 2010
Location: West Rockhill, PA
Posts: 2,233
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 337 Post(s)
Liked: 346
I spent most of yesterday playing around with FlexRAID settings. Increasing the Processes, I/O Buffer and Threads settings seemed to help but I still have some tweaking to do. I thought I had it fixed but I only delayed the memory problem. I had a validate running for nearly two hours with the total memory use never going above 8GB and modified memory staying below 100MB. I had to leave for about an hour and when I returned it was back to 96% memory in use with most of it being modified memory.

I have a scheduled verify running now and tomorrow morning a validate is scheduled to run so I don't know if I'll be able to do much testing for the next couple of days.

Right now I have the Processes at 8, I/O Buffer at 50MB and Threads at 8.
Attached Thumbnails
Click image for larger version

Name:	verify operation 37 percent.jpg
Views:	7
Size:	704.8 KB
ID:	576049  
Ken.F is offline  
post #4315 of 4316 Old Today, 08:03 AM
AVS Special Member
 
Dark_Slayer's Avatar
 
Join Date: May 2012
Posts: 2,929
Mentioned: 5 Post(s)
Tagged: 0 Thread(s)
Quoted: 381 Post(s)
Liked: 340
Quote:
Originally Posted by DotJun View Post
That's right, flexraid should be limited to 4gb due to its 32bit nature
I thought a 32 bit process was limited to 2GB. The other part of your post (flexraid clarification) Ken.F answered
Dark_Slayer is offline  
post #4316 of 4316 Old Today, 10:24 AM
AVS Special Member
 
DotJun's Avatar
 
Join Date: Dec 2012
Posts: 1,121
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 237 Post(s)
Liked: 68
Quote:
Originally Posted by Dark_Slayer View Post
I thought a 32 bit process was limited to 2GB. The other part of your post (flexraid clarification) Ken.F answered
2gb normal, 3 and 4gb when switches are flipped, which I just assume any enthusiast does when operating with 32 bit. I know, I shouldn't make assumptions.

Windows memory limit.
DotJun is online now  
Reply Home Theater Computers

Tags
Flexraid , Intel Core I5 4670k 3 4ghz Lga 1150 Quad Core Desktop Processor , Intel Core I3 4130 3 4 3 Fclga 1150 Processor Bx80646i34130 , Asrock , Asrock Z87m Extreme4 Lga1150 Intel Z87 Chipset Ddr3 Quad Crossfirex Quad Sli Sata3 Usb3 0 Microatx M , Asus , Asus Us , Asus Computer , Asus Components , Norco , Asus Computer International Direct , Flexraid Raid F , Seagate , Seagate Hard Drives , Seagate Freeagent Theater 1080p Hd Media Player Stcea201 Rk , Hitachi , Dell , Windows 7 Vista Xp Media Center Mce Pc Remote Control And Infrared Receiver , Hp Oem Window Media Center Mce Pc Remote Control And Infrared Receiver For Windows7 Vista Xp Home Pr , Intel , Intel Core I5 3570 3 4 Ghz Processor , Intel Pentium G2020 2 9ghz Lga 1155 Dual Core Desktop Processor , Intel Pentium G3220 3 0ghz Lga 1150 Dual Core Desktop Processor , Amd , Amd A10 6700 Richland 4 2ghz Socket Fm2 65w Quad Core Desktop Processor Amd Radeon Hd Ad6700okhlbox , Amd A6 5400k 3 6ghz Socket Fm2 Dual Core Desktop Processor , Amd A8 5600k 3 6ghz Socket Fm2 Quad Core Desk
Gear in this thread - 1080p by PriceGrabber.com



Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off