VMware ESXCLI Troubleshooting Commands
We’ve all been there. Server is disconnected from the vCenter and you have no idea what happened. This post will show you the basic ESXCLI troubleshooting commands in order for you to get a handle of the situation and get things back up and working in no time.
This post will be divided into several sections. The first is about reviewing logs. The other sections are basic ESXCLI commands needed to troubleshoot the problem and restore service. The last is a list of PowerCLI and ESX CI commands I use often to troubleshoot and restore host servers.
Great reference sites include VMware documentation center (VMware vSphere ESXi and vCenter Server 5 Documentation) and (VMware vSphere 6.5 Documentation Center).
To make the post manageable to read and write, I will only be focusing on ESXi 6.x and above.
VMware Log Analysis
The first thing you need to learn is where the logs are located and what each log contains. The log location is at:
/var/log
but I recommend moving them to SAN or NAS storage if you have the opportunity to do so. This would allow the logs for any host readable from any other host. For directions on how to do this, read my article on “how to change the scratch location of your host“.
Here is a list of the log files you will find in that directory. I highlighted the ones I tend to use most often when troubleshooting hosts but you will find them all useful at times.
Log Locations
| Log | Description |
| /var/log/auth.log | ESXi Shell authentication success and failure |
| /var/log/esxupdate.log | ESXi patch and update installation logs |
| /var/log/lacp.log | Link Aggregation Control Protocol logs |
| /var/log/hostd.log | Host management service logs, including virtual machine and host Task and Events, communication with the vSphere Client and vCenter Server vpxa agent, and SDK connections |
| /var/log/hostd-probe.log | Host management service responsiveness checker |
| /var/log/rhttproxy.log | HTTP connections proxied on behalf of other ESXi host webservices |
| /var/log/shell.log | ESXi Shell usage logs, including enable/disable and every command entered |
| /var/logsysboot.log | Early VMkernel startup and module loading |
| /var/log/boot.gz | A compressed file that contains boot log information |
| /var/log/syslog.log | Management service initialization, watchdogs, scheduled tasks and DCUI use |
| /var/log/usb.log | USB device arbitration events, such as discovery and pass-through to virtual machines |
| /var/log/vobd.log | VMkernel Observation events |
| /var/log/vmkernel.log | Core VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startup |
| /var/log/vmkwarning.log | A summary of Warning and Alert log messages excerpted from the VMkernel logs |
| /var/log/vmksummary.log | A summary of ESXi host startup and shutdown, and an hourly heartbeat with uptime, number of virtual machines running, and service resource consumption |
| /var/log/Xorg.log | Vide acceleration |
To view the log files, you will need to use your favorite Unix file editor. I think the most common are cat, more, less, tail and head with a little bit of grep thrown in for filtering. My favorite is “less” mainly because I am use to it and I find navigation incredibly easy and fast. I will create a separate post on how to use less in more detail but for now, you can use this post at the TheGeekStuff.com. To edit files, vi is still the standard and very easy to use.
For now, to open any log file, type the following:
less <Log File Path and Name>
Less Options for Moving Around
| Command | Description |
| / | search for a pattern which will take you to the next occurrence. |
| n | for next match in forward |
| N | for previous match in backward |
| g | go to the start of file |
| G | go to the end of file |
| q or ZZ | exit the less viewer |
Basic ESXCLI Troubleshooting Commands
The first one should be ESXTOP. This command is very useful for a quick status of anything happening on the box and is not only performance related.
esxtop
Once the screen is up, you can limit the view with several options below. Press Q to quit.
| Command | Description |
| c | switch to CPU resource utilization screen |
| m | switch to memory resource utilization screen |
| d | switch to storage (disk) adapter resource utilization screen |
| u | switch to storage (disk) device resource utilization screen |
| v | switch to storage (disk) virtual machine resource utilization screen |
| n | switch to network resource utilization screen |
| q | Quits ESXTOP |
The most popular commands are the ESXCLI command. There is a standard layout for these commands similar to Cisco IOS. Here are a few of my favorites along with the PowerShell version. One of the great things about PowerShell is there are so many ways to write your command. The list below is just one way but feel free to alter anyway you want.
Reseting ESXi Management Agents
Resetting the management agents will reconnect your VC server to your host when temporary situations disconnected them and they have not reconnected on their own. It is critical to first fix the issue that disconnected them or it will not work. Also note that when you reset your management agents on a host, if you are using NSX, you should not use the /services.sh reset option to reconnect. It is also preferible to use the console if you have access to it. The video below shows you how.
The commands are:
/etc/init.d/hostd restart
/etc/init.d/vpxa restart
services.sh restart (not if you are running NSX)
Here is a great summary in video format that will also help.
PowerCLI and CLI Commands
|
|
PowerCLI
|
ESXi Command Line
|
|
Register a VM
|
New-VM –vmfilepath “[datastore]path_to_vmx_on_datastore” –vmhostesxhost
|
vim-cmd solo/registervmpath_to_vmx_file
|
|
Unregister a VM
|
Remove-VM vm_name
|
vim-cmd
vmsvc/unregister vmid
|
|
Delete a VM
|
Remove-VM vm_name -deletepermanently
|
vim-cmd vmsvc/destroy vmid
|
|
Get a listing of VMs on a host
|
Get-VM –location esxhost
|
esxcli vm process list
vim-cmd vmsvc/getallvms
|
|
Determine if a VM has a snapshot
|
Get-VM –name vm_name | Get-Snapshot
|
vim-cmd vmsvc/get.snapshot vmid
|
|
Take a snapshot of a VM
|
Get-VM –name vm_name | New-Snapshot –name snapshot_name
|
vim-cmd vmsvc/snapshot.createvmid snapshot_name
|
|
Remove a snapshot of a VM
|
Get-VM –name vm_name | Get-Snapshot –name snapshot_name | Remove-Snapshot
|
vim-cmd vmsvc/snapshot.remove vmid
|
|
Get the current power state of a VM
|
Get-VM –name vm_name
|
vim-cmd vmsvc/power.getstate vmid
|
|
Get the uptime for a VM
|
Get-Stat -entity vm_name -stat sys.uptime.latest -MaxSamples 1
|
vim-cmd vmsvc/get.summaryvmid |grep uptimeSeconds
|
|
Power on a VM
|
Start-VM –vm vm_name
|
vim-cmd vmsvc/power.on vmid
|
|
Shutdown a VM
|
Shutdown-VMGuest –vm vm_name
|
vim-cmd vmsvc/power.shutdown vmid
|
|
Power off a VM
|
Stop-VM –vm vm_name
|
esxcli vm process kill –wworld_id
vim-cmd vmsvc/power.off vmid
|
|
Reboot a VM
|
Restart-VMGuest –vm vm_name
|
vim-cmd vmsvc/power.reboot vmid
|
|
Reset a VM
|
Restart-VM –vm vm_name
|
vim-cmd vmsvc/power.reset vmid
|
|
Upgrade VMware Tools in a VM
|
Update-Tools –vm vm_name
|
vim-cmd vmsvc/tools.upgrade vmid
|
|
Display the IP address of a VM
|
Get-VMGuestNetworkInterface –vmvm_name -guestuser guest_admin_user-guestpassword guest_admin_password
|
vim-cmd vmsvc/get.guest vmid|grep -m 1 “ipAddress = \””
|
| Put Host In or Out of Maintenance Mode |
Set-VMHost -VMHost [$vmhost] -State “Maintenance” -RunAsync
or
Get-VMHost -Name [$vmhost] | set-vmhost -State Maintenance
|
esxcli system maintenanceMode set –enable [true/false]
|
| List ESXi Hosts and Properties |
Get-VMHost
|
esxcli system [coredump, settings, stats, accounts, hostname, version, etc]
|
| vMotion VM | Get-VM VM1 | Move-VM -Destination (Get-VMHost ESXHost2) |
Not possible. Only option is to power off VM, unregister it from dead host, register it on another host, power up.
|
| Create VM |
New-VM
|
Not practical to create VM’s in CLI (but possible)
|
|
|
|
Notes:
vcenter is your vCenter Server hostname
esxhost is your ESX/ESXi hostname
datastore is the display name of your datastorevm_name is the display name of a virtual machine
path_to_vmx_file is the full path to a virtual machine’s vmx file
snapshot_name is the name given to a virtual machine snapshot
path_to_vmx_on_datastore is the path to the virtual machine’s vmx file relative to the datastore on which it resides
guest_admin_user is a user account with administrative access within a virtual machine’s guest OS
guest_admin_password is the password for the account noted by guest_admin_user
Conclusions
This is just a drop in the bucket of all the ESXCLI troubleshooting commands you can use with PowerCLI and ESXi Command line tools. The best thing to do is read through all the VMware documentation online and practice the commands in your lab. This is the best and fastest way to get use to the commands and get practice using them Before you actually need them in a customer outage.
Did I forget a common command you feel should be added? Put it in the commend section and I will add both the ESXi and PowerCLI versions to the list. Also, to get new articles delivered right to your mailbox, don’t forget to signup to my email list. Its like free training every month on new DevOps tools and knowledge!
