One of the unfortunate parts in dealing with data center technology is uncovering “gotchas.” It is unfortunate in that no matter how much you know and how much experience you have, something will always sneak up every so often and bite you in the rear. Technology changes so fast, and there are so many moving parts that it really is unavoidable in some cases. I learned of something like this last week in regards to using an AWS storage gateway to utilize AWS S3 as a backup repository within Veeam. I posted a brief write up on Public Cloud options with Veeam a little while back, and will update that content based on this finding.
We deployed AWS volume gateway for a customer so that they could utilize S3 as a backup repository within Veeam. The deployment went well and initial test backups showed that things were performing as expected. At some point after the fact, the Perform backup files health check advanced option was enabled for some jobs that targeted the S3 storage.
This seemed like an innocent enough idea and is certainly recommended in many cases to ensure the consistency of your backup files. In this case, it incurred a rather large AWS bill that month due to egress charges for bringing that data down from AWS to perform the health check locally.
One place where this can turn into a big “gotcha” is with backup copy jobs. Unlike regular backup jobs, the health check option is ENABLED by default. In the case of using an AWS gateway for copy jobs direct to S3 storage, definitely be sure to uncheck this option during job creation.
One of the big changes when it comes to Public Cloud is the consumption based pay model. Sure, there is a huge learning curve technology wise between something like vSphere administration to AWS, but on top of that businesses also need to completely change the way they think about IT spending as well. While you can think you have all your bases covered, there will always be something out there that throws you a curve-ball. Unfortunately, with Public Cloud, that curve-ball may not become apparent until it appears on a billing statement. Our customers pay us for gear and services, but also expect a certain level of expertise. While we definitely can’t (and don’t) know everything about every little nuance of the technology we deploy, we continually learn new best practices in a very dynamic landscape. In this case, definitely beware of scheduling Veeam health checks for any backup options that target Public Cloud storage!