WELCOME TO BLOG-STACK.NET

A collection of technical solutions from all corners of IT

Homepage

The Application Stack

Posts related to Microsoft software

Latest Posts on Microsoft products

The Networking Stack

All posts related to IP routing and switching

Latest Networking Posts

The Virtualization Stack

VMWare and Hyper-V related posts

info@website.com

+123 456 7890 | +123 456 7890

Automate the Stack!

Everything related to scripting, programming, and making life easier!

View Posts on Automation and Scripting

ESXi Remove All Snapshots hangs at 99%

Check and make sure that the process is actually stuck and not just taking a very long time.
To do this, follow these steps:

1. Make sure SSH service is running on the ESXi host

Navigate to configuration -> security profile -> services -> Click on Properties
esxi-services

Choose the ‘SSH’ service, ‘Options’, and click ‘Start’
esxi-ssh

Once SSH is enabled, connect to the ESXi host using your favorite ssh client.

2. Run -> vim-cmd vmsvc/getallvms
getallvms

Find the vmid of the virtual machine, in this case its 20.

3. Now run -> vim-cmd vmsvc/get.tasklist 20<- this is the # of the vm from above
gettasklist

4. Find the removeallsnapshots task # then run:

vim-cmd vimsvc/task_info 185720854 <-this # will be different for you

If you see:  “The object has already been deleted or has not been completely created” or similar, your snapshot removal is probably frozen, but in order to be sure, let’s look at the file sizes and make sure nothing is changing.

5. cd into the datastore where the vm is located.

The path is /vmfs/volumes/datastore-name/vm-name
(NOTE: tab-complete is your friend)

vmfs

6. Now Run ->  watch -d 'ls -lut | grep -E "delta|flat|sesparse"'
before-vpxa
Take notice on the file sizes and make sure they stay the same. If they are changing, the snapshot removal process is still working and I would just wait it out. Don’t risk data loss!

7. If the data file sizes doesn’t change for a long time ( 1-2 hours, just to be safe)
Restart the vpxa service on the ESXi host.
Configuration -> security profile -> services -> properties -> vpxa ->options ->restart
Note: You will lose connectivity with the host from vCenter, just reconnect after a minute.

8. Go back to the SSH terminal and re-run the command:
watch -d 'ls -lut | grep -E "delta|flat|sesparse"'
after-vpxa

You will eventually see the delta files disappear, and will be able to power on your vm with no snapshots present ! 🙂

___________________________________________________________________________________________

32 thoughts on “ESXi Remove All Snapshots hangs at 99%

  1. Thank you very much for these helpful tips, I had the same issue while removing all snapshots on one of my ESXi host and didn’t wan to to a brutal reboot.

    After restarting the VXPA service, I couldn’t start my VMs again but I was able to perform a graceful shutdown and started all the system again with no probelm at all.

  2. Awesome site. I ran out of space on a LUN due to low space. The VM crashed and I had to SVmotion servers of the LUN. When I cancelled the snapshot, I chose to Delete ALL on the snapshot. The server was stuck at 99% for a few hours. This process is spot on the money.

  3. Saved my ass last night. started consolidate at 5:30pm, and was stuck by 8pm. Ran this and left it overnight, server booted in the morning.

  4. Restart management agent. Check snapshot manager, delete any unnecessary snapshots. Browse the data store and check for orphaned files. If orphaned files exist then create another snapshot, the delete all. This should consolidate the snapshots. If you re using esxi 4 then you may be better off cloning the guest to consolidate the snapshots.

  5. I just wanted to point something out, as I just dealt with a customer that was facing this issue. He referred me to this article, as he believed he was in the same situation. However, what he failed to realize was this was running on the other datastores associated with this VM. He was only monitoring/watching the main datastore and not the others, therefore he concluded the process was hung. Once I get involved, we quickly determined it was working on the last/3rd datastore and the process not was hung.

  6. Cudos to you Rennie – this is a really excelent post! FYI, you can commit snapshot while Exchange server is running, you do not need to bring it offline. Thnx for sharing!

  7. Also, try connecting with VI client to the host were the VM is “running”. You will see that you can see the real progress, whch randomly jumps to 99%, but gets back to the real progress…

  8. Guys & Girls,

    Please exercise plenty of patience with the SNAPSHOT REMOVAL process as indicated in this HowTo article. It took almost 16 hours for my snaps to actually be deleted. I am SO glad that I was patient in this instance because we could not have afforded a corrupted VMDK on our Exchange Infrastructure. Awesome article on monitoring snapshot removal process!!!

    David Pino
    Horizon Managed Services, LLC

  9. Thank you very much for this, though i didn’t need to restart the vpxa service. the google search that landed me here at 5:20am today also pointed out the reason why my consolidation was hung at 99%. not enough space. after freeing up some real-estate on the affected datastore, your post helped me monitor the delta files and after nearly giving up hope i saw the changes. process started at 11:20pm last night and completed this morning at 7:34am.

  10. Good procedure… Thanks
    A simpler route that I use is to log directly into the ESXi host where the VM resides and view the actual snapshot removal progress in the lower tasks pane.
    There seems to be a disconnect / miscommunication between the vcenter server and ESXi host regarding the progress of large snapshot deletions / consolidation, I believe that the root cause has something to do with a timeout at one or more of the links in the chain.

  11. Thanks very much for sharing your experience in this event , It was same problem I have today with even ironically Exch-Server , Even more ironic the part which you mention (everyone and their mother had called me) is happened with my case , Note in may case the VM didn’t work till I restarted the ESXI host , Thanks again

  12. I’m currently experiencing the same thing, but in my case the delta files are not constantly changing size (I suppose because we are using thick provisioning), but rather the time stamp is changing meaning they are getting accessed/modified. After some time the delta file just disappears once it is deleted. Progress in vSphere still shows 99% – I trust when the last file is delete it will show completed (100%).

    Just before vdp_2-000001-delta.vmdk is deleted (notice the time stamp on the file):

    Every 2s: ls -lut | grep -E “delta|flat|sesparse” 2016-09-09 14:53:27

    -rw——- 1 root root 145477341184 Sep 9 14:52 vdp_2-000001-delta.vmdk
    -rw——- 1 root root 147104731136 Sep 9 12:15 vdp_3-000001-delta.vmdk
    -rw——- 1 root root 107374182400 Jun 6 2015 vdp-flat.vmdk
    -rw——- 1 root root 1099511627776 Jun 6 2015 vdp_2-flat.vmdk
    -rw——- 1 root root 1099511627776 Jun 6 2015 vdp_3-flat.vmdk
    -rw——- 1 root root 1099511627776 Jun 6 2015 vdp_1-flat.vmdk

    File vdp_2-000001-delta.vmdk is then deleted:

    Every 2s: ls -lut | grep -E “delta|flat|sesparse” 2016-09-09 14:53:29

    -rw——- 1 root root 147104731136 Sep 9 12:15 vdp_3-000001-delta.vmdk
    -rw——- 1 root root 107374182400 Jun 6 2015 vdp-flat.vmdk
    -rw——- 1 root root 1099511627776 Jun 6 2015 vdp_2-flat.vmdk
    -rw——- 1 root root 1099511627776 Jun 6 2015 vdp_3-flat.vmdk
    -rw——- 1 root root 1099511627776 Jun 6 2015 vdp_1-flat.vmdk
    Every 2s: ls -lut | grep -E “delta|flat|sesparse”

    Time stamp on file vdp_3-000001-delta.vmdk starts changing, but size remains the same:

    Every 2s: ls -lut | grep -E “delta|flat|sesparse” 2016-09-09 15:32:34

    -rw——- 1 root root 147104731136 Sep 9 15:32 vdp_3-000001-delta.vmdk
    -rw——- 1 root root 107374182400 Jun 6 2015 vdp-flat.vmdk
    -rw——- 1 root root 1099511627776 Jun 6 2015 vdp_2-flat.vmdk
    -rw——- 1 root root 1099511627776 Jun 6 2015 vdp_3-flat.vmdk
    -rw——- 1 root root 1099511627776 Jun 6 2015 vdp_1-flat.vmdk

    Regards

    John Ndambuki

  13. The snapshot removal task completed after 2hours & 20 mins without having to restart the vpxa service:

    Every 2s: ls -lut | grep -E “delta|flat|sesparse” 2016-09-09 15:54:32

    -rw——- 1 root root 107374182400 Jun 6 2015 vdp-flat.vmdk
    -rw——- 1 root root 1099511627776 Jun 6 2015 vdp_2-flat.vmdk
    -rw——- 1 root root 1099511627776 Jun 6 2015 vdp_3-flat.vmdk
    -rw——- 1 root root 1099511627776 Jun 6 2015 vdp_1-flat.vmdk

    Regards

    John Ndambuki

  14. It took about 8 hours to complete 100GB of snapshot deletion stuck with 99% for 7 hours.

    The object has already been deleted or has not been completely created – was reported but eventually the task completed with 100% after 8 hours

  15. I had to wait it out as well… The file size stayed the same on the delta file, but the time-stamp was updating. It took 3 hour for 40GB snapshot file.

  16. Hi,if we restart the service i want to make sure when the vm will be , present or before snapshot(lost the data)?

  17. What’s up,I check your new stuff named “ESXi Remove All Snapshots hangs at 99% – Blog-Stack.net” on a regular basis.Your humoristic style is awesome, keep it up! And you can look our website about powerful love spells.

Leave a Reply

Your email address will not be published. Required fields are marked *