Jobs running to a Scale-Out Repository report “No scale-out repository extents have sufficient disk space to store the backup file” while at least some of the extents have plenty of free space.
Symptom Description:
Backups are not running as expected due to a situation where the cloud backup repository was unavailable, causing the local Windows Agent to write to the local Backup Cache instead. Ordinarily this would not cause an issue and once the target is available again a Backup Cache Sync job should send them offsite.
However, because Veeam calculates the free space on a Scale Out Backup Repository (SOBR) and stores this value, as opposed to checking before starting a Backup Cache Sync operation, Veeam may not know a repository has sufficient free space to accept the cache sync job, causing the job to fail.
Here are some details on how the default Veeam logic works, excerpted from here:
This issue is related to the logic of how Veeam calculating space available on SOBR extents. Veeam caches free space information from the database. When resources for backup of a certain VM are assigned, we check the space it will take. For instance, VM is 10GB. Requested free space for its full backup will be 5GB (50% of VM size). Veeam now thinks 95GB is available on this extent. While there is at least 1 active task Veeam does not update for free space information. It uses its cache considering the size of VMs currently being backed up.
For instance, in 15 minutes we will try to assign resources for a new task, the meanwhile first one only created storage and started writing into it, having taken only 1GB for now. We may decide 99GB are available, but that’s not correct. Available space is still 95GB.
Let’s say, in half an hour new task is started, SOBR resource needs to be assigned. Active tasks are still running, so we still have 95GB free space. New VM requested size 10GB. A resource is assigned, Veeam now thinks free space is 85GB (100-5-10). The actual status will be polled only if an extent is not used by any job.
Below are the steps to identify if this issue is happening, as well as the VCC-wide fix to update the calculation method. Note: The below fix has been applied to the eSilo VCC environment as of 5/29/2020, under eSilo VCC VBR version 9.5.4.2866.
Possible symptoms of this problem may include:
Under Backup Jobs, you may first notice a Failed to Apply [Backup Policy] Error under:
Second, you may see a lapse of restore points (screenshot taken on 5/28 with last restore point on 5/22), without any RPO warning. This may indicate that the Backup Agent has taken a backup within the acceptable RPO threshold, but those recent backups are not reflecting in the Service Provider Console below. A mismatch in the machine’s RPO vs. service provider console view of RPO may indicate the Backup Cache is in use and out of sync between the host and the cloud connect server.
Logging into the Cloud Connect VBR Console, under Cloud Connect >> Last 24 Hours >> Running you may also see frequent recurring BackupCacheSync jobs start and immediately stop for the tenant.
In that same menu under Cloud Connect >> Last 24 Hours >> Success you may also see several BackupCacheSync jobs kicking off and completing with “success” every 10 minutes even though very little to no data is actually transferred:
Downloading Tenant job logs, you should also be able to find logs for the BAckup Cache Sync Job(s). Those logs may be found here, C:\ProgramData\Veeam\Backup\CloudConnectService\[TENANT]\[SUBTENANT]\BackupCacheSync_[JOB_NAME].
On the service provider side, we can see on the Cloud Gateway server, a new incoming cloud session that is almost immediately terminated.
Digging further into the Session Log (C:\ProgramData\Veeam\Backup\CloudConnectService\[TENANT]\[SUBTENANT]\Scheduler\Session.log) on the Service Provider VCC server, you may find the true root cause:
“No scale-out repository extents have sufficient disk space to store the backup file.”
Solution Steps
To address this issue you must disable the use of estimated free space on the Scale-Out Backup Repository by creating the following registry key:
Key: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\SobrForceExtentSpaceUpdate
Type: DWORD
Value: 1
This Veeam KB article covers this in more detail: https://www.veeam.com/kb2282
