ESX IP STORAGE TROUBLESHOOTING BEST PRACTICE

VMware has released a new White paper about ESXi IP storage troubleshooting.

In this paper, we:
• Describe how you can analyze packet traces to identify functional and performance issues in an
ESX IP storage environment.
• Compare packet capture alternatives, and explain why we recommend an inline optical network
tap connected to a packet capture system.
• Present the challenges of 10G packet capture, and describe key features of commercial 10G
capture solutions.
• Describe the design of an inexpensive, self-assembled 10G packet capture solution optimized for
troubleshooting that you can build relatively easily. We also describe our experience with multiple
prototypes of this design, which we have used in our ESX IP storage testbeds for NFS and iSCSI
performance for many years.
• Present examples of analyzing packet traces to solve ESX performance issues for NFSv41, software
iSCSI over IPv6, and hardware iSCSI.

ESX-IP-storage-troubleshooting.pdf

Show ESXi build details on command line

If you are logged into an ESXi per SSH you can display details about the build version using 2 simple commands.

* vmware -vl
* cat /etc/vmware/.buildInfo

Example output:

Using username "root".
The time and date of this login have been sent to the system logs.

VMware offers supported, powerful system administration tools.  Please
see www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
[root@ESXi:~] vmware -vl
VMware ESXi 6.5.0 build-5310538
VMware ESXi 6.5.0 GA
[root@ESXi:~] cat /etc/vmware/.buildInfo
GITHASH:0
CHANGE:5045323
BRANCH:vsphere65ep2
UID:201
VMTREE:/build/mts/release/bora-5310538/bora
VMBLD:release
BUILDTAG:gobuild
BUILDNUMBER:5310538
[root@ESXi:~]

ESXi show details about ramdisk usage

The command below will let you check the space that is free on the host for each of the ramdisk mount points. It also shows you the usage of inodes per ramdisk.

system visorfs ramdisk list	List the RAM disks used by the host.
~ # esxcli system visorfs ramdisk list
Ramdisk Name  System  Include in Coredumps   Reserved      Maximum       Used  Peak Used  Free  Reserved Free  Maximum Inodes  Allocated Inodes  Used Inodes  Mount Point
------------  ------  --------------------  ---------  -----------  ---------  ---------  ----  -------------  --------------  ----------------  -----------  ---------------------------
root            true                  true  32768 KiB    32768 KiB   3916 KiB   4076 KiB  88 %           88 %            8192              8192         5257  /
etc             true                  true  28672 KiB    28672 KiB    504 KiB    712 KiB  98 %           98 %            4096              1024          527  /etc
tmp            false                 false   2048 KiB   196608 KiB  29620 KiB  65016 KiB  84 %            0 %            8192              8192         3770  /tmp
hostdstats     false                 false      0 KiB  1078272 KiB  94324 KiB  99624 KiB  91 %            0 %            8192                32            5  /var/lib/vmware/hostd/stats

The comand below will give more details about a single ramdisk.

vsish -e get /system/visorfs/ramdisks/[RAMDISK-NAME]/stats

Example for root ramdisk

~ # vsish -e get /system/visorfs/ramdisks/root/stats
VisorFS ramdisk {
   Min:32 MB
   Max:32 MB
   Number of pages used:979
   Max number of pages used:1019
   Mem group ID:157
   Root inode:0
   Dump on coredump:1
   System:1
   Mount point inode:-6
   Root path:/
   First inode of ramdisk:0
   Max number of inodes:8192
   Number of allocated/initialized inodes:8192
   Number of used inodes:5263
   Max number of used inodes:8192
}

Example for tmp ramdisk

~ # vsish -e get /system/visorfs/ramdisks/tmp/stats
VisorFS ramdisk {
   Min:2 MB
   Max:192 MB
   Number of pages used:7405
   Max number of pages used:16254
   Mem group ID:1014
   Root inode:12288
   Dump on coredump:0
   System:0
   Mount point inode:8
   Root path:/tmp
   First inode of ramdisk:12288
   Max number of inodes:8192
   Number of allocated/initialized inodes:8192
   Number of used inodes:3770
   Max number of used inodes:8192
}

This will help you to troubleshoot out of disk space or out of inodes issues on a ESXi.

In one of my next posts I will go into details how to troubleshoot inode issues on a ESXi.

Time to replace my Homelab – Part 1

My current homelab contains 2x HP ML10v2, 1x HP Microserver Gen8 and 2 Switches (1 Mikrotik Cloud Switch 24 Port, Zyxcel 16 Port). This setup was sufficient for a long time. But due to the increased resource requirements of some VMware products and my activities with vROPs, Log Insight and other solutions I require more resources. Furthermore I would like to reduce from 3 to 1 server and run more activities virtually based on the lab scripts from William Lam.

Current Setup:

HostnameCPUMemoryStorageAdd-On
xmesx01eIntel® Pentium® G3240 (3.1GHz/2-core/3MB/54W)32 GB1x 240GB Intel SSD1x HP Quad Gbit NIC
xmesx02eIntel® Pentium® G3240 (3.1GHz/2-core/3MB/54W)20 GB1x 240GB Intel SSD1x HP Quad Gbit NIC
Storage (Freenas on HP Microserver Gen8):Intel® Pentium® G2020T (2.5GHz/2-core/3MB/35W)16 GB4x 2TB WD RED1x HP Smart Array P410 + 512MB Cache Modul + Battery

New Setup:

CategoryDescription
Type:HP Proliant DL380 G7  
ChipsetIntel  ®   5520 Tylersburg + Intel ICH 10
CPU TypIntel ®  Xeon ®  CPU L5630
CPU Speed2.13 GHz
Cores / Threads2 CPU(s), 4 Cores per CPU, 2 Thread(s) per Core, 16 total
L1, L2, L3 Cache32k+32k,   256k,    12288k (12 MB)
Memory32 GB DDR3 ECC 
Installed8 x 4 GB
SAS - Controller:HP Smart Array P410i Controller
Controller PCIe:1x GB LAN Broadcom PCIe Karte
2x Fiber Channel Finsar Singel Port PCIe card
Disks8x 2,5“ SAS HDDs 

The plan is to extend the machine with 8x 146GB SAS disks and additionally upgrade the memory to 144GB at least. The FC card will be removed as it will not be used in my Homelab.

If someone is interested in buying one or two HP ML10v2 please send my a message.

to be continued …

Homelab the current state

It’s time for a new update about my new Homelab. In the past I was running HP Microserver 2x N40L and 1x Gen8. All these machines are still here but no longer in use and will be for sale soon. My new Homelab contains now 2x HP ML10v2, 1x Zyxel 16 Gbit Switch, 1x Lenovo ix4-300d storage.

2x Server HP ML10 v2: HP ProLiant ML10 v2 G3240-Server (nicht Hot-Plug-fähig, 4 GB-U, B120i, 4 LFF, 350-W-Netzteil)
2x SSD adapter per Server: Kingston SNA-DC2/35 SATA DriveCarrier Einbaurahmen für 6,4 cm (2,5 Zoll) bis 8,9 cm (3,5 Zoll) schwarz
Memory: Samsung 8GB (1x 8GB) DDR3 1600MHz (PC3 12800E) 2Rx8 ECC Unbuffered Dimm Server Workstation Arbeitsspeicher RAM Memory
1x Zyxel 16 Port Gbit Switch: Zyxel GS1100-16-EU0101F Gigabit Switch (16-Port, RJ-45)
1x Lenovo ix4-300d Storage: Lenovo Iomega ix4-300d Network Storage (0TB Diskless EMEA, Marvell Armada TM XP, 1,3GHz, 4x HDD, 512MB RAM)
2x HP Quad Network adapter: HP 538696-B21 PCI-e Quad Port Gigabit Server Adapter
2x Intel SSD 240GB: Intel SSDSC2BW240H601 interner Solid State Drive 240GB schwarz
4x WD RED 2TB: WD 2TB Red interne NAS-Festplatte (8,9 cm (3,5 Zoll), 5400rpm, SATA III) WD20EFRX

xmesx01e

xmesx02e

I have installed vSphere 6 on my servers. The Lenovo storage is providing 2 iSCSI targets to fulfil the requirement of 2 shared datastores for HA. Both datastores are part of an SDRS cluster. Besides this I have created a cluster which has HA and DRS enabled. On the network side I have created 3 standard vSwitches each with 2 Gbit network adapters. vSwitch0 is for the virtual machine traffic. vSwitch1 is for iSCSI and vSwitch2 is for VMotion. Yes I know thats a bit the old school way of networking but hey thats only my Homelab so a dvSwitch with Network IO control should not be required. A simple setup is sufficient for Homelab use. Maybe in the future I’m going to change it but that will have to wait until the next vSphere release.

So whats next?

Next will be to setup vRealize Operations Manager 6.3 to improve my troubleshooting knowledge of this product. Additionally I have an idea about how to simplify the troubleshooting and improve the supportability of the product. But this is a different topic. Today I’m going to install the lastest ESXi patches and check for upgrades of my vCenter appliance.

More to come …

NetApp NFS APD issues – reduction of MaxQueueDepth

If you face APD’s in your environment you can follow the KB below to possible improve the situation.

http://kb.vmware.com/kb/2016122
https://kb.netapp.com/support/index?page=content&id=1014696

When using NFS datastores on some NetApp NFS filer models on an ESXi/ESX host, you experience these symptoms:
* The NFS datastores appear to be unavailable (grayed out) in vCenter Server, or when accessed through the vSphere Client
* The NFS shares reappear after few minutes
* Virtual machines located on the NFS datastore are in a hung/paused state when the NFS datastore is unavailable
* This issue is most often seen after a host upgrade to ESXi 5.x or the addition of an ESXi 5.x host to the environment but can also occur in vSphere 6 environment.

/var/log/vmkernel.log

NFSLock: 515: Stop accessing fd 0xc21eba0 4
NFS: 283: Lost connection to the server 192.168.100.1 mount point /vol/datastore01, mounted as bf7ce3db-42c081a2-0000-000000000000 (“datastore01”)
NFSLock: 477: Start accessing fd 0xc21eba0 again
NFS: 292: Restored connection to the server 192.168.100.1 mount point /vol/datastore01, mounted as bf7ce3db-42c081a2-0000-000000000000 (“datastore01”)
T

Additionally VMware released a new Patch for ESXi 5.5 / 6 which contains improvements of the NFS implementation which should make the ESXi more resilient to APDs.

You can find an great overview on the following sites. ESXi 5.5 Patches and ESXi 6 Patches

Besides running the latest version of ESXi it is highly recommended to apply the NetApp NFS vSphere recommendations.

Alternative Setup of a VMware Homelab

My planed new Homelab requires a lot of space due to the fact of the size of the components. But what if someone would like to have a Homelab but does not have the space in the office. One option would be to run a fully virtualized Homelab but this is limited to the resources of the computer/laptop. A more costly option would be to use multiple INTEL NUC and a storage system.

Shopping list:

2x Intel NUC i5: Intel NUC6i5SYH
2x Samsung 32GB Memory: SAMSUNG 32GB Dual Channel Kit 2 x 16 GB 260 pin DDR4 2133 SO-DIMM (2133Mhz, PC4-17000, CL15) passend für alle Intel “Skylake” Notebooks mit DDR4 Standard
2x Samsung M2 SSD 250GB: Samsung MZ-N5E250BW 850 EVO interne SSD 250GB (SATA) grün
1x QNAP NAS 4 Bay: QNAP TS-453A-8G-NAS 4-Bay Intel Celeron Braswell N
4x WD Red 3TB: WD 3TB Red interne NAS-Festplatte (8,9 cm (3,5 Zoll) SATA) WD30EFRX
1x Netgear Managed Switch 8 port: Netgear ProSafe GS108T (8-Port Gigabit Smart Managed Switch 8 x 10/100/1000 – desktop)

You can use the standard ESXi (5.5 or 6) images provided from the VMware web site and install it using a USB stick.

As soon as I have all hardware available I’m going to document the complete setup on my web site.

Homelab upgrade and Options to install a SSD into the HP Microserver Gen8

I started to plan the upgrade my current homelab from 1x HP Microserver Gen8, 2x HP Microserver N40L and a Netgear 24 port switch. The only part which I will keep is the HP Microserver Gen8 it has already 16GB of memory and a HP SmartArray P410 with 512MB Cache module incl. battery. This machine will run my infrastructure machines like a tiny vCenter. As work horses I plan to use HP ML10v2 due to the fact that they are cost efficient and they support up to 32GB of memory.

Parts list which I will use in my future homelab:

1x Server HP Microserver Gen8: HP ProLiant MicroServer (Gen8, G1610T, 1P, 4 GB-U, B120i, SATA-Server)
2x Server HP ML10 v2: HP ProLiant ML10 v2 G3240-Server (nicht Hot-Plug-fähig, 4 GB-U, B120i, 4 LFF, 350-W-Netzteil)
2x SSDs per Server: Mushkin MKNSSDRE1TB Reactor 7mm SSD, 1TB
2x SSD adapter per Server: Kingston SNA-DC2/35 SATA DriveCarrier Einbaurahmen für 6,4 cm (2,5 Zoll) bis 8,9 cm (3,5 Zoll) schwarz
2x Disks per Server: HGST Deskstar NAS 3TB 6Gb/s SATA 7200rpm 24×7 RV S
Switch: TP-Link TL-SG3424 Pure-Gigabit L2 Managed Switch (24x 10/100/1000Mbps RJ45-Ports inkl. 4 kombinierter SFP-Anschlüsse, lüfterloses Passivkühlkonzept)
Memory: Samsung 8GB (1x 8GB) DDR3 1600MHz (PC3 12800E) 2Rx8 ECC Unbuffered Dimm Server Workstation Arbeitsspeicher RAM Memory

There are at least 2 options to install SSDs into the HP Microserver Gen8.

Option 1 is to connect a single SSD to the SATA port on the mainboard an place it where normally the DVD drive should be installed. That’s the cheapest method and might be not the best one.

Option 2 will use a adapter to map the 2.5″ SSD into a 3.5″ tray. I personally would use the Kingston SNA-DC2/35 adapter.