Trigger a User event from the VCSA command line

Disclaimer: This is not officially supported by VMware, please use at your own risk.

Recently I faced the challenge with a script running on a VCSA which executes some checks and in case something gets detected it has to trigger a vCenter alarm.

My solution is to log a user event which will be captured by an vCenter alert definition.

1. vCenter alert definition

as trigger manually enter (type): vim.event.GeneralUserEvent

2. The script which will log the user event

#!/usr/bin/python

import sys
sys.path.insert(0, '/usr/lib/vmware/site-packages/')
from pyVim.connect import SmartConnect
from pyVmomi import vim
import ssl
import atexit

s = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
s.verify_mode = ssl.CERT_NONE

si = SmartConnect(host="localhost", user="administrator@vsphere.local", pwd='VMware1!', sslContext=s)

content = si.RetrieveContent()
ds = content.rootFolder

log = si.content.eventManager.LogUserEvent(entity=ds, msg="Issue detected")

Result:

The resulting alert can be catched by vROPs and trigger further events/ tickets.

vCenter notification send push message using Prowl to iOS devices

I would like to receive push messages for triggered alerts from my vCenter. There is the possibility to execute a command when a alert has been triggered so I decided to write a script which will use Prowl to send push notification to my mobile apple device. I used Prowl already for other devices like my Homematic home automation system. It’s easy to use with an simple curl command. You can find a lot of good examples available by using Google.

Sources and Services:

VMware Documentation

Push notification service

You have to create an account on www.prowlapp.com and generate a new API key. The new API key has to be added to script (replace PLACE_YOUR_API_KEY_HERE with your API key).

Script:

This script is really basic and is only a proof of concept. I will extend and improve it over time.

root@vcenter [ ~ ]# mkdir bin
root@vcenter [ ~ ]# cd bin
root@vcenter [ ~/bin ]# vi alert.sh
#!/bin/bash
#set -x

value="$VMWARE_ALARM_ALARMVALUE"
if [ $$value == "red" ]; then
priority=2
else
priority=0
fi
app="$VMWARE_ALARM_NAME"
event="$VMWARE_ALARM_EVENTDESCRIPTION"
description="$VMWARE_ALARM_TARGET_NAME"
apikey=PLACE_YOUR_API_KEY_HERE

curl https://prowl.weks.net/publicapi/add -F apikey=$apikey -F priority=$priority -F application="$app" -F event="$event" -F description="$description"

Edit vCenter alert definition:

To receive the push notification you have to install the prowl app on your iOS device and login from the app to prowl.

Thats basically all what is required for vCenter push notifications on your iPhone.

Get all guest IPs of all your Virtual Machines using Powershell

This is a small powershell script to get all Guest IPs from all virtual machines

$VCENTER="10.1.1.3"
Connect-VIServer -Server $VCENTER

$vms = Get-VM | Sort

$file = "c:\ip_list.txt"

foreach ($vm in $vms){
	
	foreach($ip in $vm.Guest.IpAddress){
		Write-Host "$vm $ip"
		"$vm $ip" | Out-File -FilePath $file -Append

	}

}

Disconnect-VIServer -Server $VCENTER -Confirm:$false -Force:$true

Upload file to webdav using Powershell

This is small powershell script to upload a file from your local disk to a webdav server. It is also proxy capable.

#Complete path of the file to be uploaded
$file = "d:\test_file.txt"
 
#URL without the last "/"
$url  = "https://YOUR-SERVER/webdav"
$proxy = "http=PROXY-SERVER:PORT"  
 
#User and Pwd for Webdav Access
$user = "USERNAME"
$pass = "PASSWORD"
 
$url += "/" + $file.split('\')[(($file.split("\")).count - 1)]

# Set binary file type
Set-Variable -name adFileTypeBinary -value 1 -option Constant 
 
$objADOStream = New-Object -ComObject ADODB.Stream
$objADOStream.Open()
$objADOStream.Type = $adFileTypeBinary
$objADOStream.LoadFromFile("$file")
$buffer = $objADOStream.Read()
$objXMLHTTP = New-Object -ComObject MSXML2.ServerXMLHTTP
$objXMLHTTP.setProxy(2, $proxy,"")
$objXMLHTTP.Open("PUT", $url, $False, $user, $pass)
$objXMLHTTP.send($buffer)

Monitor System Logs with Logwatch

I’m using the tool Logwatch to get a daily log report from all my servers by email.

Install Logwatch:

apt-get update
apt-get install logwatch

Config file:

/usr/share/logwatch/default.conf/logwatch.conf

To simplify the access to the config file I use a symlink.

cd /etc/logwatch
ln -s /usr/share/logwatch/default.conf/logwatch.conf

Configuration (parameters which I have changed):

#Output = stdout
Output = mail
#To make Html the default formatting Format = html
Format = html
MailTo = YOUR-EMAIL-ADDRESS

Cronjob:

crontab -e
30 0 * * * /usr/sbin/logwatch

Based on this configuration you will receive a nice daily overview report.

Collect ESXi stats with powershell and send it to Graphite server

I was not quite happy with the statistics from the vCenter. It is not possible to get an overview overall my ESXi servers. :-( Someone pointed me to the tool Graphite that this is a cool solution to visualize such kind of statistics. So I decided to give it a try.

I created and virtual machine running Centos and Graphite as target for my collected statistics. I will post an How to later.

Furthermore I had to create an powershell script which collects the stats of each ESXi in my cluster, transform it a graphite compatible format and transfer it to the graphite server.

#vCenter settings
$vCenter = "VCENTER-IP"
$user = "USERNAME"
$password = "PASSWORD"
$cluster = "YOUR-CLUSTER"
#Graphite server 
$remoteHost = "GRAPHITE-SERVER-IP"

#Socket to send results to Graphite server	 
$socket = new-object System.Net.Sockets.TcpClient($remoteHost, 2003)
$stream = $socket.GetStream()
$writer = new-object System.IO.StreamWriter $stream

Write-Host "Connected"
#Connect to vCenter
Connect-VIServer -Server $vCenter -User $user -Password $password 

#Get all ESXi hosts from Cluster
$esxhosts = Get-VMHost -Location $cluster | Sort

#Collect stats foreach ESXi server and bring it in a Graphite compatible format
foreach ($esxName in $esxhosts){

	$allstats = Get-Stat -Entity (Get-VMHost $esxName) -MaxSamples 1 -Realtime -Stat cpu.usage.average,disk.usage.average,net.usage.average | Sort
	Write-Host $esxName
	foreach($stat in $allstats){
		#Get Timestamp of stats and convert to UNIX timestamp
		$time = $stat.Timestamp
		$date = [int][double]::Parse((Get-Date -Date $time -UFormat %s))
		#Filter only average stats (Stats for CPU's are available foreach CPU and as average of all CPU's)
		$instance = $stat.Instance
		if($instance -eq [DBNull]::Value){
			#create a base for the graphite tree
			$base = "dc.vmware.prod."
			#remove the .usage.average
			$type = (($stat.MetricId).Replace(".usage.average",""))
			#remove the domain from the esxi name
			$name = (($esxName.Name).Replace(" ","")).Replace(".your-domain.de","")
			$value = $stat.Value
			#build the graphite compatible string
			$result = "$base$name.$type $value $date"
			#Console output just for testing
			Write-Host "Sent Result: $result"
			#send result to graphite server
			$writer.WriteLine($result)
			$writer.Flush()
		}
	}
	Write-Host " "
}
## Close the streams
$writer.Close()
$stream.Close()
#disconnect from vcenter
Disconnect-VIServer -Server $vCenter -Confirm:$false -Force
Write-Host "Done"

Below is a screenshot of an Graphite graph displaying the CPU average usage of all ESXi servers.

That’s it :-)

HomeLab and nested ESXi server

After an short weekend with a minimal of sleep … I’m done my Homelab is up and running. Furthermore I have created an template of an nested ESXi (virtualized ESXi). These nested ESXi are perfect for testing of Powershell scripts which I use to initial configure an new ESXi. Next steps which I have planned is to migrate my existing scripts to Powershell 2.0 modules and additionally extend the useability of it. So hopefully it will be possible to enter the IP into an GUI and select per SelectBox different configurations and than just hit configure. :-)

Change HA settings for virtual machines using Powershell

Using this small script it is possible to change the HA settings for multiple virtual machines in an HA cluster.

Example:

– multiple virtual machines all with Name XXX-console

HA restart policy possible values:

-HARestartPriority:Low
-HARestartPriority:High
-HARestartPriority:Disabled
-HARestartPriority:Medium
-HARestartPriority:ClusterRestartPriority

Connect-VIServer -Server VCENTER-IP -User USER -Password PASSWORD

$Cluster = "CLUSTERNAME"
$vms = Get-VM -Location $Cluster -Name "*-console" | sort

foreach ($vm in $vms) {
	Write-Host $vm
	$result = $vm | Set-VM -HARestartPriority:Low -Confirm:$false
}

Disconnect-VIServer -Server VCENTER-IP -Force:$true -Confirm:$false

Configure local syslog folder on all your ESXi servers

########################################################################################################
# Steps to use the script:                                                                             #
# 1st: create a folder on an shared datastore                                                          # 
# 2nd: adjust the content of the variable $FolderName to the name of the folder created in step 1      #
# 3rd: adjust the content of the variable $datastore to the name of the shared datastore               #
# 4th: adjust vCenter IP/Username/Password to match your settings                                      #
########################################################################################################

Connect-VIServer -Server VCIPADDRESS -User USERNAME -Password PASSWORD

$esxihosts = Get-VMHost
$FolderName = "esxi_syslog"
$datastore = "NAMEofSAN"

foreach ($esxi in $esxihosts) {
	$hostname = $esxi.Name.split(".")[0]
	$value = "[$datastore] $Foldername/$hostname.log"
	Write-Host $value
	Set-VMHostAdvancedConfiguration -VMHost $esxi -Name "Syslog.Local.DatastorePath" -Value $value
	Clear-Variable hostname -ErrorAction SilentlyContinue
	Clear-Variable value -ErrorAction SilentlyContinue
}
Disconnect-VIServer -Server VCIPADDRESS -Force -Confirm:$false

VMware Tools Status report

Script to create a report about the VMware Tools status:

Sample Output:

Total VMs: 527
Total powered on: 474
Total powered off: 53
ToolsOK: 140
ToolsOld: 227
ToolsNotInstalled: 106
DisConnect from Virtual Center

Set-PSDebug -Strict
$VIServer = "VCENTERIP"
Connect-VIServer -server $VIServer -User USERNAME -Password PASSWORD

$vms = Get-VM | Sort Name
$on = 0
$toolsOk = 0
$toolsOld = 0
$toolsNotInstalled = 0
foreach ($vm in $vms){
	if ($vm.PowerState -like "PoweredOn"){
		$vmview = $vm | Get-View
		$toolsstatus = $vmview.Guest.ToolsStatus
		$toolsrunningstatus = $vmview.Guest.ToolsRunningStatus
		$toolsversionstatus = $vmview.Guest.ToolsVersionStatus
		$on = $on+1
		if ($toolsstatus -like "toolsOk"){
			$toolsOk = $toolsOk + 1
		}
		if ($toolsstatus -like "toolsOld"){
			$toolsOld = $toolsOld + 1
		}
		if ($toolsstatus -like "toolsNotInstalled"){
			$toolsNotInstalled = $toolsNotInstalled + 1
		}
	}
}
$count = $vms.Count
$off = $count - $on
Write-Host "Total VMs: $count" 
Write-Host "Total powered on: $on"
Write-Host "Total powered off: $off"
Write-Host "ToolsOK: $toolsOk"
Write-Host "ToolsOld: $toolsOld"
Write-Host "ToolsNotInstalled: $toolsNotInstalled"
# DisConnect to Virtual Center
Write-Host "DisConnect from Virtual Center"
Disconnect-VIServer -Server $VIServer -Confirm:$False