VMware ESXCLI Troubleshooting Commands

ESXCLI Troubleshooting COmmands

We’ve all been there.  Server is disconnected from the vCenter and you have no idea what happened.  This post will show you the basic ESXCLI troubleshooting commands in order for you to get a handle of the situation and get things back up and working in no time.

This post will be divided into several sections.  The first is about reviewing logs.  The other sections are basic ESXCLI commands needed to troubleshoot the problem and restore service.  The last is a list of PowerCLI and ESX CI commands I use often to troubleshoot and restore host servers.

Great reference sites include VMware documentation center (VMware vSphere ESXi and vCenter Server 5 Documentation) and (VMware vSphere 6.5 Documentation Center).

To make the post manageable to read and write, I will only be focusing on ESXi 6.x and above.

VMware Log Analysis

The first thing you need to learn is where the logs are located and what each log contains.  The log location is at:

/var/log

but I recommend moving them to SAN or NAS storage if you have the opportunity to do so.  This would allow the logs for any host readable from any other host.  For directions on how to do this, read my article on “how to change the scratch location of your host“.

Here is a list of the log files you will find in that directory.  I highlighted the ones I tend to use most often when troubleshooting hosts but you will find them all useful at times.

Log Locations

LogDescription
/var/log/auth.logESXi Shell authentication success and failure
/var/log/esxupdate.logESXi patch and update installation logs
/var/log/lacp.logLink Aggregation Control Protocol logs
/var/log/hostd.logHost management service logs, including virtual machine and host Task and Events, communication with the vSphere Client and vCenter Server vpxa agent, and SDK connections
/var/log/hostd-probe.logHost management service responsiveness checker
/var/log/rhttproxy.logHTTP connections proxied on behalf of other ESXi host webservices
/var/log/shell.logESXi Shell usage logs, including enable/disable and every command entered
/var/logsysboot.logEarly VMkernel startup and module loading
/var/log/boot.gzA compressed file that contains boot log information
/var/log/syslog.logManagement service initialization, watchdogs, scheduled tasks and DCUI use
/var/log/usb.logUSB device arbitration events, such as discovery and pass-through to virtual machines
/var/log/vobd.logVMkernel Observation events
/var/log/vmkernel.logCore VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startup
/var/log/vmkwarning.logA summary of Warning and Alert log messages excerpted from the VMkernel logs
/var/log/vmksummary.logA summary of ESXi host startup and shutdown, and an hourly heartbeat with uptime, number of virtual machines running, and service resource consumption
/var/log/Xorg.logVide acceleration

To view the log files, you will need to use your favorite Unix file editor.  I think the most common are cat, more, less, tail and head with a little bit of grep thrown in for filtering.  My favorite is “less” mainly because I am use to it and I find navigation incredibly easy and fast.  I will create a separate post on how to use less in more detail but for now, you can use this post at the TheGeekStuff.com.  To edit files, vi is still the standard and very easy to use.

For now, to open any log file, type the following:

less <Log File Path and Name>

Less Options for Moving Around

CommandDescription
/search for a pattern which will take you to the next occurrence.
nfor next match in forward
Nfor previous match in backward
ggo to the start of file
Ggo to the end of file
q or ZZexit the less viewer
  

Basic ESXCLI Troubleshooting Commands

The first one should be ESXTOP.  This command is very useful for a quick status of anything happening on the box and is not only performance related.

esxtop

Once the screen is up, you can limit the view with several options below.  Press Q to quit.

CommandDescription
cswitch to CPU resource utilization screen
mswitch to memory resource utilization screen
dswitch to storage (disk) adapter resource utilization screen
uswitch to storage (disk) device resource utilization screen
vswitch to storage (disk) virtual machine resource utilization screen
nswitch to network resource utilization screen
qQuits ESXTOP

The most popular commands are the ESXCLI command.  There is a standard layout for these commands similar to Cisco IOS.  Here are a few of my favorites along with the PowerShell version.  One of the great things about PowerShell is there are so many ways to write your command.  The list below is just one way but feel free to alter anyway you want.

Reseting ESXi Management Agents

Resetting the management agents will reconnect your VC server to your host when temporary situations disconnected them and they have not reconnected on their own.  It is critical to first fix the issue that disconnected them or it will not work.  Also note that when you reset your management agents on a host, if you are using NSX, you should not use the /services.sh reset option to reconnect.  It is also preferible to use the console if you have access to it.  The video below shows you how.

The commands are:

/etc/init.d/hostd restart

/etc/init.d/vpxa restart

services.sh restart (not if you are running NSX)

Here is a great summary in video format that will also help.

PowerCLI and CLI Commands

 
PowerCLI
ESXi Command Line
Register a VM
New-VM –vmfilepath “[datastore]path_to_vmx_on_datastore” –vmhostesxhost
vim-cmd solo/registervmpath_to_vmx_file
Unregister a VM
Remove-VM vm_name
vim-cmd
vmsvc/unregister vmid
Delete a VM
Remove-VM vm_name -deletepermanently
vim-cmd vmsvc/destroy vmid
Get a listing of VMs on a host
Get-VM –location esxhost
esxcli vm process list
vim-cmd vmsvc/getallvms
Determine if a VM has a snapshot
Get-VM –name vm_name | Get-Snapshot
vim-cmd vmsvc/get.snapshot vmid
Take a snapshot of a VM
Get-VM –name vm_name | New-Snapshot –name snapshot_name
vim-cmd vmsvc/snapshot.createvmid snapshot_name
Remove a snapshot of a VM
Get-VM –name vm_name | Get-Snapshot –name snapshot_name | Remove-Snapshot
vim-cmd vmsvc/snapshot.remove vmid
Get the current power state of a VM
Get-VM –name vm_name
vim-cmd vmsvc/power.getstate vmid
Get the uptime for a VM
Get-Stat -entity vm_name -stat sys.uptime.latest -MaxSamples 1
vim-cmd vmsvc/get.summaryvmid |grep uptimeSeconds
Power on a VM
Start-VM –vm vm_name
vim-cmd vmsvc/power.on vmid
Shutdown a VM
Shutdown-VMGuest –vm vm_name
vim-cmd vmsvc/power.shutdown vmid
Power off a VM
Stop-VM –vm vm_name
esxcli vm process kill –wworld_id
vim-cmd vmsvc/power.off vmid
Reboot a VM
Restart-VMGuest –vm vm_name
vim-cmd vmsvc/power.reboot vmid
Reset a VM
Restart-VM –vm vm_name
vim-cmd vmsvc/power.reset vmid
Upgrade VMware Tools in a VM
Update-Tools –vm vm_name
vim-cmd vmsvc/tools.upgrade vmid
Display the IP address of a VM
Get-VMGuestNetworkInterface –vmvm_name -guestuser guest_admin_user-guestpassword guest_admin_password
vim-cmd vmsvc/get.guest vmid|grep -m 1 “ipAddress = \””
Put Host In or Out of Maintenance Mode
Set-VMHost -VMHost [$vmhost] -State “Maintenance” -RunAsync
or
Get-VMHost -Name [$vmhost] | set-vmhost -State Maintenance
esxcli system maintenanceMode set –enable [true/false]
List ESXi Hosts and Properties
Get-VMHost
esxcli system [coredump, settings, stats, accounts, hostname, version, etc]
vMotion VM Get-VM VM1 | Move-VM -Destination (Get-VMHost ESXHost2)
Not possible.  Only option is to power off VM, unregister it from dead host, register it on another host, power up.
Create VM
New-VM
Not practical to create VM’s in CLI (but possible)
 
 
 

Notes:

vcenter is your vCenter Server hostname
esxhost is your ESX/ESXi hostname
datastore is the display name of your datastore
vm_name is the display name of a virtual machine
path_to_vmx_file is the full path to a virtual machine’s vmx file
snapshot_name is the name given to a virtual machine snapshot
path_to_vmx_on_datastore is the path to the virtual machine’s vmx file relative to the datastore on which it resides
guest_admin_user is a user account with administrative access within a virtual machine’s guest OS
guest_admin_password is the password for the account noted by guest_admin_user

Conclusions

This is just a drop in the bucket of all the ESXCLI troubleshooting commands you can use with PowerCLI and ESXi Command line tools.  The best thing to do is read through all the VMware documentation online and practice the commands in your lab.  This is the best and fastest way to get use to the commands and get practice using them Before you actually need them in a customer outage.

Did I forget a common command you feel should be added?  Put it in the commend section and I will add both the ESXi and PowerCLI versions to the list.  Also, to get new articles delivered right to your mailbox, don’t forget to signup to my email list.  Its like free training every month on new DevOps tools and knowledge!

0 Shares

TJ Totland

Todd "TJ" Totland is a computer and network engineer working for IBM. He is certified as a MCSE, MCT, CNE, and CNA with vast experience in many technologies used in businesses today. TJ has designed, built, and managed hundreds of different types of computer and network systems for large and small customers since 1990. He is a subject Matter Expert in Cloud Technologies and has vast knowledge in VMware products.