Have you ever experienced the joy associated with overcoming a trying (or annoying) IT issue and the feeling of peace that comes with running a successful test before you leave for the night….knowing full well that this particular problem is behind you, that you can cross this one of the fix list. Later in the evening after dinner, even though YOU KNOW the problem is resolved, you VPN into the network anyway, just to see the success message so as to stroke your ego and prove to yourself once again that you are awesome, you are brilliant, and you are indeed “all that”!
Then as you smugly open the admin console, your ego comes crashing back down to earth in a flaming mess and as it falls, it passes your aggravation as it skyrockets to heights you didn’t even know existed.
That was me last week….
As I was trying to do is setup Veeam to backup to an EMC Data Domain appliance. Easy right? Yes, it should be. I believe the primary reason it wasn’t is because this organization was not licensed for Veeam Enterprise, thus I had to setup the DD as a CIFS share and to be honest with you, I didn’t set it up in such a way as to achieve the optimal compression and deduplication of the backup jobs so the full backups filled all of the available DD disk space. I ended up moving all the backup files to another disk, setup a new DD CIFS share, a new Veeam repository with the correct settings, and set the backup jobs to use it, making sure they too were setup according to Veeam best practices for DD CIFS shares. I ran a successful test backup, left the site feeling good (finally) fully confident that my backup problems were a thing of the past.
Sometime right after dinner, I got the notification that the backup failed! WHAT!!!! How can that be? Logging into the console, I saw that the backup failed due to an incorrect user name or password!?!?!?!?!?!? I just did a test 3 hours ago!!
Seeking to prove the Veeam server had lost its mind, I attempted to login to the vSphere client with the same credentials. I angrily pressed the keys and slammed <Enter> as though I were the IT equivalent of LeBron James doing a “get out of my face” tomahawk dunk! I was surprised to see that I was unable to login to the vCenter server, the error was virtually the same as that seen within the Veeam console….I had just missed my tomahawk dunk and Veeam had drawn the charge.
Logging into the Windows-based vCenter server, I tried restarting the vCenter server service but it wouldn’t start. So I rebooted the vCenter server, yet the vCenter server service persisted in not starting. Opening Task Manager, I saw that several Java processes in use by other VMware services were consuming every scrap of RAM, leaving none for the vCenter server service.
To reign in Java consumption, I adjusted the wrapper.conf configuration files for the following VMware services to scale back the maximum Java heap size:
- vCenter Inventory Service
- wrapper.conf file stored within the installation_disk\Program Files\VMware\Infrastructure\Inventory Service\conf directory
- vSphere Profile-Driven Storage
- wrapper.conf file stored within the installation_disk\Program Files\VMware\Infrastructure\Profile-Driven Storage\conf directory
- vSphere Web Client
- wrapper.conf file stored within the installation_disk\Program Files\VMware\Infrastructure\vSphereWebClient\server\bin\service\conf directory
- VMware Log Browser
- wrapper.conf file stored within the installation_disk\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf directory
Knowing that this environment likely will not grow beyond 100 VMs, I felt comfortable cutting the JVM wrapper.java.maxmemory values by 50%. After making these changes, I also shut down the vCenter server and gave the VM a little more virtual RAM. When booted back up, the vCenter server service started, I could login using the vSphere client, AND upon relaunching the Veeam backup job, I was able to successfully backup VMs.
Final Thought
With the move from Windows-based servers to vCenter virtual appliances, this type of problem should die a natural death. But until all Windows-based vCenter servers are decommissioned, you may run into a similar issue so be mindful Java’s RAM usage on Windows-based vCenter servers and their potential impact on Veeam backup jobs.
