Veeam Backup error related to CBT

Description:

Backup jobs keep failing or taking longer time to finish due to failed to create change tracking time stamp for virtual disk file.

Error:CBT error

Resolution:

We have restarted each hyper-v host.

updated hyper-v integration tools on hyper-v hosts

migrated few VMs from the Volume1 to new Volume 3

still no luck

I have found the Veeam article https://www.veeam.com/kb1934 , which helped to resolve our issue.

To reset changed block tracking, we have performed the following steps from article:

  • Select the option to disable changed block tracking on all Hyper-V hosts in the cluster.  To do this, right-click on each host in the cluster, select Manage Volumes and then check the box to disable changed block tracking.

User-added image

  • Run sc stop VeeamFCT, in an elevated command prompt, on each Hyper-V host in the cluster to ensure the CBT service is stopped.
  • Find, export and then delete the following registry key on all Hyper-V hosts in the cluster:  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VeeamFCT\Parameters\CTDB
  • Go back to the Manage Volumes for each host and uncheck the disable changed block tracking option.  Try the job after this.  Allow a couple of job runs before ensuring CBT is working.
  • If the issue persists, reboot each Hyper-V host in the cluster.  This will typically involve migrating VMs from one host to another and rebooting the host once it does not own any VMs.  Repeat these steps for all Hyper-V hosts in the cluster.

Hope the above steps helps.

Existing tape unable to recognise the data

Scenario:

Untitled picture

In Commvault, After we import number of tapes in library, the Monthly or DR Tapes were unable to recognise the data or contents inside the tapes.

 

Resolution:

We have implemented the below steps to resolve this issue:

 

Step1: Logon to the Commvault console

 

Step2: Browse to the Storage Resources > Libraries > Media By Location and Media In Library

Untitled picture2

Step3: Right Click each Media and Delete it. After delete you should see this icon Untitled picture3

Untitled picture4

Step4: Right Click the each Media and Click the Discover option to re-discover.

Untitled picture5

Discover Job in Job Controller should show as below screenshot:

Untitled picture6

After the Tape discovered, you will see the tape icon as Untitled picture7

HOW TO Enable, Disable, Configure and Maintain the SIOC in VMWare

As i was studying for the vmware exams, i have found the below details from VMware website as useful on SIOC. Hope it helps others as well.

Reference Link: https://kb.vmware.com/s/article/1022091

Enabling Storage I/O Control

To enable SIOC:

  1. Select a datastore in the vSphere Client inventory and click the Configuration tab.
  2. Click Properties.
  3. Under Storage I/O Control, select Enabled.
  4. Click Close.
Note: This setting is specific to the datastore and not to the host.

If you experience problems with SIOC or if the number of hosts connected to the datastore has changed since enabling SIOC:

  1. Disable SIOC and save the changes by clicking OK.
  2. Enable SIOC and save the changes.

Determining if the threshold value has been modified

To determine if the threshold value has been modified:
  1. Select a datastore in the vSphere Client inventory and click the Configuration tab.
  2. Click Properties.
  3. Under Storage I/O Control, click Advanced.
  4. Check if the value is 30ms. If it is not 30, reset it to the default value of 30.

Ensuring virtual machines have disk shares assigned according to their importance

By default, all virtual machines have the same number of shares and IOPS limit. IOPS are the number of I/O operations per second. By default, IOPS are unlimited. If these defaults are not changed, then I/O control does not prioritize virtual machines.

To see the shares of all the virtual machines on the cluster, choose the cluster, click Resource Allocation, then click Storage.

To change the vDisk shares and limit:

  1. Choose a virtual machine in the vSphere Client inventory.
  2. Click the Summary tab and click Edit Settings.
  3. Click the Resources tab and click Disk.
    • Choose a virtual hard disk from the list and click the Share column to select the relative amount of shares to allocate to the virtual machine (Low, Normal, or High).
    • You can also click Custom and enter a user-defined share value.

  4. Click the Limit – IOPS column and enter the upper limit of storage resources to allocate to the virtual machine.
  5. Click OK.

Note: The above process is used to set the resource consumption limits of individual vdisks in a virtual machine even when SIOC is not enabled. These settings are specific to the individual guest, and not the host, although they are used by SIOC.

Enabling and disabling SIOC logging on the host

These logs are used for troubleshooting purposes. If you are sending the support logs to VMware, enable SIOC logging before collecting the logs. VMware recommends that you disable logging after the troubleshooting activity is complete.

To enable logging:

  1. Click Host > Configuration.
  2. In the left pane, click Software and then click Advanced Settings.
  3. In the Misc section, select the Misc.SIOControlLogLevel parameter.
  4. Set the value to 7 for complete logging.

    For example:

    Min value: 0 (no logging)
    Max value: 7

    After changing the log level, you see these changes logged in the/var/log/vmkernel logs:

    May 27 07:41:10 wdc-tse-h53 vmkernel: 76:14:16:03.334 cpu7:4110)Config: 297: “SIOControlLoglevel” = 1, Old Value: 0, (Status: 0x0)

  5. Perform the action that is failing or repeat the procedure to replicate the observed issue.
To disable logging:
  1. Click Host > Configuration.
  2. In the left pane, click Software and then click Advanced Settings.
  3. Set the Misc.SIOControlLogLevel value to 0.
  4. After changing the log level, you see these changes logged in the /var/log/vmkernel logs.

    May 27 07:41:31 wdc-tse-h53 vmkernel: 76:14:16:24.521 cpu4:4111)Config: 297: “SIOControlLoglevel” = 0, Old Value: 1, (Status: 0x0)

Note: SIOC log files are saved in /var/log/messages. In ESXi 5.0 or later versions, the SIOC log files are saved in /var/log/storagerm.log.

Run this command to stop and start SIOC on ESX;

/etc/init.d/vmware-late {start|stop|status|restart}

Run this command to stop and start SIOC on ESXi:

/etc/init.d/storageRM {start|stop|status|restart}

HOW to Synchronize clocks in vsphere network

Synchronize ESXi Clocks with a Network Time Server

NOTE: Before you install vCenter Server or deploy the vCenter Server Appliance, make sure all machines on your esxi hosts or existing cluster were synchronized with NTP server.

Steps to update or check the ESXi NTP server settings:

  • Start the vSphere Client, and connect to the ESXi host.

 

  • On the Configuration tab, click Time Configuration.

 

  • Click Properties, and click Options.

 

  • Select NTP Settings.

 

  • Click Add. In the Add NTP Server dialog box, enter the IP address or fully qualified domain name of the NTP server to synchronize with.

 

  • Click OK.

Note: Check to see if the host time synchronizes with the NTP server.

 

How to configure the Network Details on ESXi host

Steps to configure the Network settings in ESXi host:

Logon to the console of the ESXi host through idrac, Cisco KVM or HP ilo.

Untitled picture

Press F2 > Enter username and password that you setup at the installation.

Untitled picture1

Browse to Configure Management network setting and Press Enter >

Untitled picture2

Enter Vlan and Click OK >

Untitled picture3

Enter IP address details and Click OK >

Untitled picture4

Select IPv6 and Disable it.

Untitled picture5

Enter DNS and Click OK >

Untitled picture6

Enter Custom DNS and Click OK >

Untitled picture7

press ESC to restart the management agents.

Untitled picture8

 

Unable to remove warning message in cluster in VMware vCenter Server HA failover in progress – staled records

Warning message:

HA Failover is in progress on cluster and on datacentre: 1 VMs restart, 0 VMs failed…

 

Cause:

This issue occurs when a HA failover event occurs in the cluster, which triggers the warning message. This locks the warning message and prevents it from being removed.

 

Solution:

I have similar warnings on 3 clusters in our vcenters, the below steps did helped me. I already tried to reset to green or clear these alarms from cluster level but no luck.

  1. Disable HA at the cluster level.
  2. Enable HA at the cluster level.