Unable to power on Nutanix CVM in ESXi 6.0 due to Virtual Nested Paging error

As I was unable to 'google' a solution to this issue, nor was the issue a known condition within Nutanix's support team, I'm writing this article as a potential life-saver for anyone that may run into this very specific error condition down the road.

Environment:
ESXi 6.0 on Nutanix NX-8035-G5 hosts
Acropolis 5.0.3.1

Condition:  
After running ESXi updates on an ESXi host within this cluster, the CVM (cluster virtual machine) failed to power on with the following error message:

Power On virtual machine
The virtual machine cannot be powered on because virtual nested paging is not compatible with PCI passthru in VMware ESX 6.0.0.

*Note that it says 'passthru' instead of the proper spelling 'passthrough.'  This threw off google searches like whoa.

After contacting Nutanix support, it was initially thought that the VM itself was having PCI Passthrough issues, so he played around with those settings to no avail.   We also tried upgrading the VM hardware compatibility level of the CVM from 8 to 11, which also did not help.

It eventually occurred to me that we have the vhv-enable="TRUE" setting enabled on the esxi hosts within this cluster, as a remedy for vmotion issues & VMs with older hardware levels.  This is set within the ESXi hosts's /etc/vmware/config file.  After deleting this line from that file, and without rebooting the host, the CVM then immediately powered back on.  I could confirm this as the fix by powering down the VM, re-enabling that setting, and then attempting and failing to start the CVM again.

As I was concerned that disabling this host-level setting would impact some of our customer's VMs, I re-enabled this setting on the ESXi host and then negated it on the CVM itself.  So:

(on the esxi host in question)

1. Edit the .vmx file for the CVM.  Do not use the GUI, as I found that it did not save the modification.
   vi /vmfs/volumes/[path-to-nutanix-local-storage]/[path-to-cvm-home]/[cvm-name].vmx
   or
   vi /vmfs/volumes/59bb934a-d42388b0-3ae0-0cc47a9ba4c2/ServiceVM_Centos//ServiceVM_Centos.vmx
2. Add vhv-enable = "FALSE" to the bottom of the file.  Check to see that vhv.enable = "TRUE" is set, if so, change to "FALSE".
3. Save the file.
4. Start the VM.
5. #winning

You should review this setting on all hosts within your cluster, as it's unlikely that you'll have vhv.enable set on just one host.  

Gabe's Pancakes of Glory

I made this today for my family and we were all blown away by the awesomeness.  If you're in the mood for rich, thick, sweet pancakes, then you're in the right place.

The key to this recipe is the Coconut oil as a substitute for the typically-american Butter.  You have enough fat & cholesterol with the eggs and milk, so why add more?  If you don't have coconut oil, go get some before you start.  You won't regret it.  It really is the key ingredient, more than you might suspect.  The ingredients stated below should feed 4-5 people just fine.  I tend to multiply it by 1.5 to feed my family of 6.  Your mileage may vary.  Your happiness won't.

Dry Ingredients: 
2 Cups Flour (white or wheat, your choice)
1 Cup Sugar
1 Tsp Salt
2 Tblsp Baking Powder

Wet Ingredients:
1.5 Cups of Milk
2 Eggs
4 Tsp Vanilla
0.5 Cups Melted Coconut Oil
Sprayable Olive Oil (for the skillet)

Instructions:

Start by putting a pot of coffee (or tea) on.  If it's cool outside, open the windows a bit to get a little breeze (not much.)  It is important to experience the hot coffee and cool breeze at the same time.  But not too cool, because you don't want to freeze your batter.. or your butt.

In a large mixing bowl, combine the dry ingredients.  You don't have to sift the flour.. who does that, really?  Just combine and mix well with a wire whisker.  (Not from a metal cat.  Although if you have a metal cat, I pity you.)   Begin preheating your skillet to around 325 F.

In a smaller mixing bowl, combine the liquid ingredients (minus the coconut oil) and them aside.  Take a sip of your coffee.  Take a deep breathe of your cool, fresh air coming in the window before you proceed.  This is most important: don't forget to take a deep, slow breath.

You might notice that the Coconut oil is listed in the wet ingredients, but in it's room-temperature state (below 78 F), it is solid.  Microwave to the rescue!  Heat it for 45-60 seconds to liquify it.  You don't want it molten, but you do want it a bit hot.  If it's not warm enough, the cold milk/eggs will cause it to thicken too quickly.  Gabe3000 prefers Glass Measuring Cups!

At this point you should have a small bowl of liquid gold coconut oil, a bigger bowl of the liquid ingredients, and a bigger bowl of the dry.  Combine the oil with the dry, stir enough to spread it around (don't waste your time stirring forever.)  You don't want to add the oil to the liquid bowl because the shock of the hot/cold will solify too quickly.. Next, combine the liquid with the dry/oil mix.  Stir slowly, not rapidly.  You want to mix, but not turn your batter into soup.  Stir out the big clumps and make sure there's nothing stuck to the bottom or sides of the bowl.

You should notice that the hot oil plus the cold milk are beginning to thicken.  This is good.  Take another sip of your coffee and another deep breath of fresh air.  Spray your skillet with Olive oil (not canola or that horrid 'butter flavored oil.'  If you're feeling frisky, put some coconut oil on the skillet instead of Olive oil.  Now you're rockin!

After thought:  If you're using a non-stick pan, such as what with Teflon coating, then you likely won't need to use oil on the skillet.  You also won't likely have living birds in your house, because cooking with Teflon-coated stuff kills birds.

You probably can figure out the rest from here.   However, in case you've never cooked pancakes before, use a 1/4 cup scooper to lay the batter out on the hot & oiled skillet.  Don't rely on time, but keep your eyes on the pancakes.. when the edges begin to pull up from the sides and the bubbles in the middles stop popping, you're probably ready to flip it.  Flip it.  Give it a few minutes to brown the other side, then stick on a plate.

And don't put butter on them.  You just had an amazing, butter-free experience.. don't ruin it by putting butter on it.  Just put syrup (natural is the best) on them, and enjoy.  And take your time.  You don't want to rush your morning.

After thought #2:  If you have extras, put them in a zip-lock bag and stick'm in the fridge.  Eat as a snack later.  You won't believe how delicious they taste on their own.

WSS 3.0 - Search Engine Errors

So I was setting up a sharepoint (WSS 3.0) site for a customer, and started getting the following errors in the event log:




Log Name:      Application
Source:        Windows SharePoint Services 3 Search
Date:          2/23/2012 10:48:05 AM
Event ID:      2436
Task Category: Gatherer
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      xxxx.xxx.local
Description:
The start address cannot be crawled.


Context: Application 'Search index file on the search server', Catalog 'Search'


Details:
The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured correctly.   (0x80041200)


AND



Log Name:      Application
Source:        Windows SharePoint Services 3 Search
Date:          2/23/2012 10:48:05 AM
Event ID:      2424
Task Category: Gatherer
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      xxx-xxx.cds.local
Description:
The update cannot be started because the content sources cannot be accessed. Fix the errors and try the update again.


Context: Application 'Search', Catalog 'index file on the search server Search'





I looked around and thought this was related to DB permissions.. there are plenty of google's that indicate this, but I've done this often enough to know how to setup the permissions on service accounts.

What it turned out to be, and what I could not seem to find anyone else that discovered the same (which is why I'm posting this blog) is that it was due to the site being unavailable via HTTP.  I had locked down the site to only allow HTTPS.. you can see in the above error messages that the protocol being used is STS3.. not STS3S.  


If you review the following link, you'll see something like this:

The STS3 Protocol

The STS3 protocol is used for crawling SharePoint content without having to have URL links that lead to every possible content item in the web site. This protocol is used to index WSS 3.0 server farms (which MOSS 2007 is layered on top of ) STS3 protocol achieves this by using the Site Data Web service to determine all the content in your SharePoint site.

Some examples:
sts3://www.yourdomain.com/
sts3://www.yourdomain.com/teamsites/
sts3://portal.yourdomain.com/

The protocol handlers (that ship out-of-the-box with MOSS 2007) translate the content source url from sts3:// into a web service call sts3://www.yourdomain.com is actually crawled on the web front end by using the web service http://mysite.yourdomain.com/_vti_bin/sitedata.asmx ( the STS3 SOAP call is made using HTTP protocol on TCP Port 56737 )

Note, that if you are using SSL on your Site host headers, and you do not expose the pages through non-SSL URLs, the STS3 protocol should not be used, and you should change your content source to use the STS3S protocol instead.

So yeah, of course it's trying to access the site via non-SSL, and is failing.  Not user permissions related AT ALL.


The fix for this was to go in to my Central Administration > Operations > Alternate Access Mappings, and look at my URLs.  Yep, my default zone was http://xxx.local.  I changed it to be https://xxx.local.  (I also edited my HOSTS file to point to the internal IP for this website, as it would normally resolve to the public IP which is unrouteable from the server itself.)  


Voila, search errors went away and all is well in the land.  I hope this helps someone else.