ESXi Build With 16.568 TB, 16 GB RAM, 6 core CPU - Page 4 - AVS Forum
Forum Jump: 
Reply
 
Thread Tools
post #91 of 112 Old 06-14-2013, 08:27 PM
Member
 
tcs2tx's Avatar
 
Join Date: Feb 2005
Location: Chicago, IL
Posts: 188
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 11
Andy-Steb - Are you running your management network (192 168.0.3) on a different subnet? I've commented before in an other thread that I've read on another site that it is best to dedicate a NIC to the management network and put it on a different subnet. If that is what you've done, could you post a screenshot of your pfsense settings? I have the extra NiC but I have hesitated playing with the network settings in pfsense out of fear the family would be without Internet for too long while I was mucking around with the subnet masking.

Thanks.
tcs2tx is offline  
Sponsored Links
Advertisement
 
post #92 of 112 Old 06-14-2013, 08:47 PM
AVS Special Member
 
Puwaha's Avatar
 
Join Date: Jan 2003
Posts: 1,124
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 33
Quote:
Originally Posted by BruceD View Post

Do you have a build thread for this setup ?

If not could you describe the component pieces of this setup ?

Which motherboard, which CPU, which SATA JBOD card, any other cards, etc ?

Sounds like a very efficient setup. When did you assemble and get it running, and which version of VMware ?

Thanks

Sorry I missed this earlier... I posted my config here: http://www.avsforum.com/t/1471777/esxi-wanted-any-gurus-on-here/30#post_23348963
Puwaha is offline  
post #93 of 112 Old 06-15-2013, 05:31 AM
AVS Special Member
 
Andy_Steb's Avatar
 
Join Date: Jan 2005
Location: New Orleans
Posts: 1,016
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 20 Post(s)
Liked: 152
Quote:
Originally Posted by tcs2tx View Post

Andy-Steb - Are you running your management network (192 168.0.3) on a different subnet? I've commented before in an other thread that I've read on another site that it is best to dedicate a NIC to the management network and put it on a different subnet. If that is what you've done, could you post a screenshot of your pfsense settings? I have the extra NiC but I have hesitated playing with the network settings in pfsense out of fear the family would be without Internet for too long while I was mucking around with the subnet masking.

Thanks.

As of right now, I'm on the same subnet. I set it up this way for future use. The Intel dual nic was dirt cheap so i installed it while I had the server apart. I really wasn't planning on subnet masking. I was thinking more along the lines of dual nics in the client computer, one on each subnet.

Andy_Steb is offline  
post #94 of 112 Old 06-16-2013, 01:20 AM
Senior Member
 
tj21's Avatar
 
Join Date: Mar 2000
Location: Prague, Czech Republic
Posts: 323
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I used to run virtualized file server. Here is the trick how you can present local disks as raw devices to one VM (running the file server): http://fojta.wordpress.com/2010/05/02/rdm-raw-disk-mapping-with-local-disks/. Note: this was written three years ago.

Today I would not do it. There are following problems:

- Single Point of Failure: you probably have quite a lot of (valuable) data on the file server. If you have problem with the the server hardware (especially with the disk controller) you cannot access the data (and in worse case there could be data corruption). If you need to reboot the server (when you upgrade the hypervisor, or HW) again no access to your data, your firewall is not working so no internet access, etc.
- Running a lot of disk in standard case makes lot of noise, heat and has high power consumption. The heat was the biggest problem for me as during summer my ventilated closet got very hot.

So what am I using today?

I have dedicated Synology DS713+ and Iomega IX4-200d storage arrays. Especially the Synology is great device - quiet, efficient, cool, can run apps, act as file server and obviously also as shared storage for ESX (NFS or iSCSI). Then I have couple very quite, low consumption, diskless servers that act as ESX cluster (with vMotion, DRS, High Availability ...). I understand that this is mostly overkill for file serving but I am an IT professional who tests various products on that rig. One node is Shuttle SZ68R5 with 32GB RAM, extra 2 port Intel NIC - booting from USB flash disk and datastores are presented as iSCSI from Synology.

As said Synology makes great devices of various size - from 1 disk up to 24 all with the same firmware (DSM). Some are expandable so you can grow and do not have to make all the investment up front. You can run apps on it (mail server, DHCP, DNS, video surveillance, web servers, ...). And they are really easy to use.

Tomas
tj21 is offline  
post #95 of 112 Old 06-16-2013, 03:16 AM
Advanced Member
 
balky's Avatar
 
Join Date: Oct 2008
Posts: 822
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 37 Post(s)
Liked: 41
Quote:
Originally Posted by tj21 View Post

I used to run virtualized file server. Here is the trick how you can present local disks as raw devices to one VM (running the file server): http://fojta.wordpress.com/2010/05/02/rdm-raw-disk-mapping-with-local-disks/. Note: this was written three years ago.

Today I would not do it. There are following problems:

- Single Point of Failure: you probably have quite a lot of (valuable) data on the file server. If you have problem with the the server hardware (especially with the disk controller) you cannot access the data (and in worse case there could be data corruption). If you need to reboot the server (when you upgrade the hypervisor, or HW) again no access to your data, your firewall is not working so no internet access, etc.
- Running a lot of disk in standard case makes lot of noise, heat and has high power consumption. The heat was the biggest problem for me as during summer my ventilated closet got very hot.

So what am I using today?

I have dedicated Synology DS713+ and Iomega IX4-200d storage arrays. Especially the Synology is great device - quiet, efficient, cool, can run apps, act as file server and obviously also as shared storage for ESX (NFS or iSCSI). Then I have couple very quite, low consumption, diskless servers that act as ESX cluster (with vMotion, DRS, High Availability ...). I understand that this is mostly overkill for file serving but I am an IT professional who tests various products on that rig. One node is Shuttle SZ68R5 with 32GB RAM, extra 2 port Intel NIC - booting from USB flash disk and datastores are presented as iSCSI from Synology.

As said Synology makes great devices of various size - from 1 disk up to 24 all with the same firmware (DSM). Some are expandable so you can grow and do not have to make all the investment up front. You can run apps on it (mail server, DHCP, DNS, video surveillance, web servers, ...). And they are really easy to use.

Tomas

For the price of the Synology DS713+ I would prefer to build two robust storage / file servers that will out-perform the Synology from almost any standpoint...
While I won't recommend the use of a virtual machine as your principal storage / file server, investing more than $1K on proprietary hardware isn't an ideal thing in my opinion for a home setup... (even for a small business)

A NAS built out of an Antec 300 case with 8 x 2T HDD, 128GB SSD for OS install, a 6 core AMD CPU and 16G of RAM will definitely not cost up to 50% of the Synology DS713 NAS and will out-perform the Synology NAS on almost anything.

In my opinion, the bottom line is determining what you really want to achieve and being able to draw the line in terms of expectations versus investment.
balky is offline  
post #96 of 112 Old 06-23-2013, 06:48 PM
Advanced Member
 
ncarty97's Avatar
 
Join Date: Mar 2006
Posts: 746
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 6 Post(s)
Liked: 16
bryansj,

Are you using the stock CPU fan? The one that came with my 8320 is quite loud, so I think I'm going to have to invest in a new one.
ncarty97 is offline  
post #97 of 112 Old 06-23-2013, 06:52 PM
Advanced Member
 
ncarty97's Avatar
 
Join Date: Mar 2006
Posts: 746
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 6 Post(s)
Liked: 16
Quote:
Originally Posted by tj21 View Post


- Single Point of Failure: you probably have quite a lot of (valuable) data on the file server. If you have problem with the the server hardware (especially with the disk controller) you cannot access the data (and in worse case there could be data corruption). If you need to reboot the server (when you upgrade the hypervisor, or HW) again no access to your data, your firewall is not working so no internet access, etc.
- Running a lot of disk in standard case makes lot of noise, heat and has high power consumption. The heat was the biggest problem for me as during summer my ventilated closet got very hot.

Most people are just putting large amounts of video/music/tv on their file servers (at least around this site). Not that you don't want that available, but having it go down for a day or two due to a hardware issue isn't exactly a major crisis. Plus, whether you virtualize or just run something like WHS on a box, unless you are planning to put your drives into multiple machines, you still have the same single point of failure issue. And if I really need any of the data on my flexraid array, I can just pull that drive out and pop into into any windows machine.

I'm loving virtualization so far. My Win7 VM has been great for remote access or having it do some handbrake conversions while I use my main rig for other stuff. Can't wait until I have my plex server up.
ncarty97 is offline  
post #98 of 112 Old 06-24-2013, 05:16 AM - Thread Starter
AVS Special Member
 
bryansj's Avatar
 
Join Date: Feb 2004
Location: Atlanta, GA
Posts: 6,270
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 38 Post(s)
Liked: 172
Quote:
Originally Posted by ncarty97 View Post

bryansj,

Are you using the stock CPU fan? The one that came with my 8320 is quite loud, so I think I'm going to have to invest in a new one.

No. I bought the one in my signature. Another thing to note about the fan noise with a stock Asrock motherboard is that the BIOS default for the CPU fan is full on. I didn't hit acceptable sound levels until I found that setting and lowered it.

bryansj is offline  
post #99 of 112 Old 06-24-2013, 05:39 AM
Advanced Member
 
ncarty97's Avatar
 
Join Date: Mar 2006
Posts: 746
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 6 Post(s)
Liked: 16
Thanks. I had read about that, but can't remember if I changed that setting or not.
ncarty97 is offline  
post #100 of 112 Old 06-27-2013, 09:10 PM
Advanced Member
 
mrted46's Avatar
 
Join Date: Jun 2006
Posts: 831
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 10
I finally installed esxi and I am confused on how to pass the ibm hba m1015 to a VM. Can anyone guide me through it?

I created a VM for WHS, but when I try to install WHS it only sees the datastore I created on the SSD.

In Configuration tab -> Advanced settings I am seeing "Host does not support passthrough configuration"
mrted46 is offline  
post #101 of 112 Old 06-27-2013, 09:51 PM
AVS Special Member
 
Andy_Steb's Avatar
 
Join Date: Jan 2005
Location: New Orleans
Posts: 1,016
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 20 Post(s)
Liked: 152
Do you have AMD-Vi or VT-d enabled in system bios?

Andy_Steb is offline  
post #102 of 112 Old 06-27-2013, 10:42 PM
Advanced Member
 
mrted46's Avatar
 
Join Date: Jun 2006
Posts: 831
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 10
Quote:
Originally Posted by Andy_Steb View Post

Do you have AMD-Vi or VT-d enabled in system bios?

I do not see option for either. Unless I am missing it?


I reinstalled esxi and now I get this:

"no devices currently enabled for passthrough"

Then I click configure and I get this:

"This device is passthrough capable but not running in pssthrough mode"

I get the above message for IBM M1015 and my intel dual nic card
mrted46 is offline  
post #103 of 112 Old 06-28-2013, 05:38 AM - Thread Starter
AVS Special Member
 
bryansj's Avatar
 
Join Date: Feb 2004
Location: Atlanta, GA
Posts: 6,270
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 38 Post(s)
Liked: 172
In the BIOS look for IOMMU for AMD based boards and enable it. IIRC, VT-d would be for Intel only.

See page 56 here. It is under the North Bridge Config.

bryansj is offline  
post #104 of 112 Old 06-28-2013, 06:19 AM
Advanced Member
 
mrted46's Avatar
 
Join Date: Jun 2006
Posts: 831
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 10
Quote:
Originally Posted by bryansj View Post

In the BIOS look for IOMMU for AMD based boards and enable it. IIRC, VT-d would be for Intel only.

See page 56 here. It is under the North Bridge Config.

I enabled IOMMU and reinstalled esxi and I get the same thing, where it says "This device is passthrough capable but not running in passthrough mode"

Also, I have virtualization enabled (by default):

Secure Virtual Machine
When this option is set to [Enabled], a VMM (Virtual Machine Architecture)
can utilize the additional hardware capabilities provided by AMD-V. The
default value is [Enabled]. Conf guration options: [Enabled] and [Disabled].


Also, Esxi sees all my hard drives the dual NIC card, I can create datastore for the HDDs and select which NIC to use for LAN but can't pass them down.

I am at a loss, but I was before starting smile.gif
mrted46 is offline  
post #105 of 112 Old 06-28-2013, 06:22 AM - Thread Starter
AVS Special Member
 
bryansj's Avatar
 
Join Date: Feb 2004
Location: Atlanta, GA
Posts: 6,270
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 38 Post(s)
Liked: 172
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010789

While you are there pay attention to this:
"Note: When the device is assigned, the virtual machine must have a memory reservation for the full configured memory size."

You have to pre-allocate all your RAM to any VM with a pass-thru device assigned. That means all that RAM is assigned to that VM and isn't part of the RAM pool for other VMs.

bryansj is offline  
post #106 of 112 Old 06-28-2013, 06:51 AM
Advanced Member
 
mrted46's Avatar
 
Join Date: Jun 2006
Posts: 831
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 10
Quote:
Originally Posted by bryansj View Post

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010789

While you are there pay attention to this:
"Note: When the device is assigned, the virtual machine must have a memory reservation for the full configured memory size."

You have to pre-allocate all your RAM to any VM with a pass-thru device assigned. That means all that RAM is assigned to that VM and isn't part of the RAM pool for other VMs.

I will try to assign all the RAM to the VM I am trying to pass down and see what happens. I am a little skeptical because even before I create a VM, on the host, it is informing me that I can't pass the nic or m1015 down because it is not running in passthrough mode.

Unfortunately, I can't try this until I get home tonight.

---
Configuring pass-through devices
To configure pass-through devices on an ESX host:

Select an ESX host from the Inventory of VMware vSphere Client.

Note: If you have a chipset with VT-d, when you click Advanced Settings in vSphere Client, you can select the devices that are dedicated to the VMDirectPath I/O.

In the Configuration tab, click Advanced Settings. The Pass-through Configuration page lists all available pass-through devices.


At this step is where I see the message. All my devices are seen as not able to pass-through or not configured to pass-through but are capable.
mrted46 is offline  
post #107 of 112 Old 06-28-2013, 07:03 AM
Advanced Member
 
ncarty97's Avatar
 
Join Date: Mar 2006
Posts: 746
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 6 Post(s)
Liked: 16
Quick question, but how difficult is it to move a datastore from one drive to another?

I ask because I am having some issues with my WHS2011 VM. In addition to the backup service and clean up service stopping running regularly, it can't seem to set folder permissions properly. It just gets stuck at the point that it is changing the permissions for the user. It seems to somewhat work in that I can then access the folders from the client PC's (despite the wizard still running), but then I can't see them when I remote login. I had installed it to a 45GB datastore on my SSD. One thing I noticed when I did this (with the trick to install on a smaller drive than 160GB) was that it did not create a seperate partition for the data and dumped the server folders on the 'C' drive.

So I was thinking, I still have my 160GB drives in the case (though not hooked up) from my old build. I was thinking of using one as a datastore to install it. It will make a C and D drive at 60GB and 100GB respectively, then I could just move the C portion back over to the SSD.

Anyone try that before? Any issue?
ncarty97 is offline  
post #108 of 112 Old 06-28-2013, 07:29 AM
AVS Special Member
 
Andy_Steb's Avatar
 
Join Date: Jan 2005
Location: New Orleans
Posts: 1,016
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 20 Post(s)
Liked: 152
Quote:
Originally Posted by mrted46 View Post

I will try to assign all the RAM to the VM I am trying to pass down and see what happens. I am a little skeptical because even before I create a VM, on the host, it is informing me that I can't pass the nic or m1015 down because it is not running in passthrough mode.

Unfortunately, I can't try this until I get home tonight.

---
Configuring pass-through devices
To configure pass-through devices on an ESX host:

Select an ESX host from the Inventory of VMware vSphere Client.

Note: If you have a chipset with VT-d, when you click Advanced Settings in vSphere Client, you can select the devices that are dedicated to the VMDirectPath I/O.

In the Configuration tab, click Advanced Settings. The Pass-through Configuration page lists all available pass-through devices.


At this step is where I see the message. All my devices are seen as not able to pass-through or not configured to pass-through but are capable.

Can you give us a screen shot?

Once your in the advanced settings you need to hit EDIT. A new window will pop up listing all the pci divices.
From here you need to place a check mark next to the controller and Nic. Hit OK, then reboot the Esxi host.

Andy_Steb is offline  
post #109 of 112 Old 06-28-2013, 08:02 AM
Advanced Member
 
mrted46's Avatar
 
Join Date: Jun 2006
Posts: 831
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 10
Quote:
Originally Posted by Andy_Steb View Post

Can you give us a screen shot?

Once your in the advanced settings you need to hit EDIT. A new window will pop up listing all the pci divices.
From here you need to place a check mark next to the controller and Nic. Hit OK, then reboot the Esxi host.

I will provide a screenshot tonight. I hit EDIT and I see all the devices but I can't hit check mark, it is greyed out and that's where I see the message on the bottom explaining why I can't select them.

This happens for both the Intel NIC and the M1015, so it has me thinking it may be a BIOS issue. I can try to see if there is a BIOS update on the web too, but I hate updating BIOS unless I have to or want the new features.
mrted46 is offline  
post #110 of 112 Old 06-28-2013, 07:48 PM
Advanced Member
 
mrted46's Avatar
 
Join Date: Jun 2006
Posts: 831
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 10
Well I got it working but couldn't tell you how. I re installed esxi yet again and this time it worked...
mrted46 is offline  
post #111 of 112 Old 07-12-2013, 09:47 AM
Advanced Member
 
mrted46's Avatar
 
Join Date: Jun 2006
Posts: 831
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 10
:facepalm: I am a mess!

I went to rackmount my new esxi build and I forgot to remove my usb drive and in the process I have crushed it...back to re-installing it lol

I have a question, what are the advantages of installing esxi on a usb stick rather than a ssd within the case?

RIP jump drive
mrted46 is offline  
post #112 of 112 Old 07-12-2013, 11:45 AM - Thread Starter
AVS Special Member
 
bryansj's Avatar
 
Join Date: Feb 2004
Location: Atlanta, GA
Posts: 6,270
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 38 Post(s)
Liked: 172
The footprint of ESXi is tiny and it would get loaded to RAM at boot. Why bother with a 1GB partition when you could just flash drive it. One option for internal mounting would be to take a USB header to USB female port and keep it attached directly to your motherboard. I had a couple dual USB bracket mounts that I could have used.

bryansj is offline  
Reply Home Theater Computers

User Tag List

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off