Before going ahead, we need to understand the rules/points that are very important in order to decide the backup preferences in AOAG configuration. Below are the important points you should consider during designing backup strategy of availability databases.

• Regular Full Backups are not allowed from secondary replicas. Only COPY_ONLY full backups are allowed on secondary replicas. That means you cannot run regular Full backup from any of the secondary replica. Remember that copy-only backups do not impact the log chain or clear the differential bitmap.
• Differential backups are not supported on secondary replicas. If you must run differential backups for your availability databases, you should not choose backup preferences as secondary replicas.
• Only regular Log backups are supported on secondary replicas. COPY_ONLY log backups are not supported on secondary replicas.
• Secondary replica must be in SYNCHRONIZED or SYNCHRONIZING state to backup a secondary database.
• Log backup chain is supported across all replicas regardless of where the log backup is taken (primary or secondary replicas) or the mode of the replication (asynchronous or synchronous).
• In a distributed availability group, backups can be performed on secondary replicas in the same availability group as the active primary replica, or on the primary replica of any secondary availability groups.

How SQL Server Decides Which Replica will execute Backup – Backup Preferences ?

It’s little complex process to determine where backup jobs should run.  If you are confused about this question, then keep reading to get the answer.

To decide on which replica, you should run your backup job depends on your Backup Preferences setting. To address this, configure the availability group to specify which availability replicas where you would prefer backups to be performed.

If you choose backup preferences as secondary only then you can create jobs on only secondary replicas or if you choose to run it on primary replica, then you can configure it on primary replica only. But if you decided to go with default option that is preferred secondary then you should schedule backup jobs on all replicas.  Although, Microsoft suggests to create scripted backup-jobs for every availability database on every server instance that hosts an availability replica that is a candidate for performing backups.

Suppose you have chosen secondary only preference and you have 3 secondary replicas then question might arise how database engine will decide that on which replica backup should run. Database engine decides where should run backup based on preferred backup replica settings and backup priority settings. I have explained about these settings later in this article.

AOAG (AlwaysOn Availability group) has a function named sys.fn_hadr_backup_is_preferred_replica  to determine which replica is set as preferred backup replica.  If the out of this function returns 1 it means current instance is the preferred backup replica and backup will run on this instance. If returns 0, it means backup will not run on that replica.  We can use this function in a backup script to determine job should run on which replica, so only one of the scheduled jobs actually proceeds to the backup stage.

In the event of a failover, none of the scripts or jobs needs to be modified. Also, if you reconfigure an availability group to add an availability replica, managing the backup job requires simply copying or scheduling the backup job. If you remove an availability replica, simply delete the backup job from the server instance that hosted that replica. Microsoft has given a sample code logic for this function that can be used to develop backup script.

If sys.fn_hadr_backup_is_preferred_replica( @dbname ) <> 1
BEGIN
-- If this is not the preferred replica, exit (probably without error).
END
-- If this is the preferred replica, continue to do the backup.

If you use the Maintenance Plan Wizard to create a given backup job, the job will automatically include the scripting logic that calls and checks the sys.fn_hadr_backup_is_preferred_replica function.

If you want to check the preferred backup replica for any availability database, you can get it by running below command. If the output of this function returns 1 it means current instance is the preferred backup replica and backup will run on this instance. If returns 0, it means backup will not run on that replica.

--Get the backup preference replica for availability database TechYaz.
SELECT sys.fn_hadr_backup_is_preferred_replica ('TechYaz');
GO

I have executed same function on my secondary replica. You can see the output is showing as 1, it means this replica is set to run backups.

Configure Backup of Availability Databases to Run only from Secondary Replicas using GUI

1. Connect to the server instance that hosts the primary replica, and click the server name to expand the server tree. You would not be able to change Backup Preferences from any secondary replicas. If you open the AOAG name property, you will find all options greyed out.
2. Expand the Alwayson High Availability node and the Availability Groups
3. Click the availability group whose backup preferences you want to configure, and select the Properties.
4. In the Availability Group Properties dialog box, select Backup Preferences You can see backup preference is set to Prefer Secondary, now we need to change it to Secondary only option. The details about each type of backup preference is given in the screenshot and AOAG backup preference property window.

5. There are 4 backup preferences in AOAG that are given below with their capabilities. Choose any as per your requirements. Now I will add one more replica to this availability group as per our requirement and set backup preference to secondary only so that backups always run from either of the both secondary replicas. It is not mandatory for you to add additional replica.

6. Use the Replica backup priorities grid to change the backup priority of the availability replicas. Backup Priority (Lowest=1, Highest=100) specifies your priority for performing backups on this replica relative to the other replicas in the same availability group. In my case backup will run on the replica that has priority 60% and if this will not be available then backup will run on the replica that has priority of 50%. Backup will never run on primary replica irrespective of backup priority settings because we have chosen to run backups only on secondary replica.
   
   We have also an option Exclude Replica to exclude any replica where you don’t want to run backups. This is useful for remote availability replicas to which you never want to run backups there.
   You should click on checkbox for your respective replica name for which you want to exclude the backup as shown in above screenshot.

7. Click on OK button to apply the changes.

Configure SQL Server Backup in AOAG to Run with default backup preference using T-SQL

I have shown you how to change backup preference setting to run backups from secondary replicas using GUI. We can do the same using T-SQL commands as well. Here i will show you how to change backup preference using T-SQL. As we have changed the backup preference to secondary only option where backups will never run from primary replica. Now we can revert this change to default option using T-SQL. Prefer Secondary is default option in which backups run from secondary replica and if only primary replica is online then it will run from primary replica only.

Connect to the server instance that hosts the primary replica and run below command.

--Choose backup preference as per your choice and pass that value in AUTOMATED_BACKUP_PREFERENCE = { PRIMARY | SECONDARY_ONLY| SECONDARY | NONE }
ALTER AVAILABILITY GROUP AG_AOAGName SET (AUTOMATED_BACKUP_PREFERENCE = SECONDARY)
GO

Now your backup preference would be set to Prefer Secondary and backups will run from primary replica if only primary replica will be online.

Related Articles:

• Fix: Secondary Replica Stucked in to Resolving state
• Fix: Connectivity Issue after AOAG Automatic Failover
• Understanding SQL Server Recovery Models
• Fix: Listener Issue Error 19471

I hope you like this article. Please follow us on our facebook page and on Twitter handle to get latest updates.

The post Understanding Backup Preferences for AlwaysOn Availability Group Databases appeared first on Techyaz.com.

I have also explained another aspect of this error in attached article. If your issue is not fixed using this article, you can have a look at this one where i got the same Listener Error 19471 because Kerberos Status was showing ‘The handle is invalid’.

Microsoft SQL Server Error 19471: Create failed for Availability Group Listener ‘Listener Name’.

AOAG Listener was failed to create during its configuration and displayed SQL Server error 19471. This error might have multiple reasons. Here i will fix this issue in step by step manner. The full error details are given below.

The screenshot of this error is given below.

When i checked the AlwaysOn High Availability folder on any of the replica, everything was working fine. AOAG was configured successfully and working fine but listener was not configured due to above error. You can see this in below screenshot.

You can see there is no listener showing in their respective folder of the AOAG configuration in above screenshot. I tried creating Listener name from “Add Listener…” wizard but failed with same error. Let’s move to next section where we will fix this error.

Solution

I did some research on this error code and found a solution suggested by Microsoft in attached MSDN article. I followed step by step process given in this article. High level approach to fix this solution using this method is given below.

• Create a Client Access Point (CAP) for the AOAG listener in Failover Cluster Manager.
• Bring CAP Online.
• Repair the CAP, if it is not coming ONLINE.
• Once CAP comes online, remove it from failover cluster manager.
• Add listener to AOAG in SSMS.

You might have a question why we are removing CAP after adding, repairing and bring it online. Creating CAP and bringing it online is to just verify that everything is ok or not. If CAP will not come online then there might be some issue at certain level that needs to be fixed first before creating listener. Let’s go ahead with above given steps to fix this issue.

I launched failover cluster manager and clicked on Roles tab from left side pane. Here you can see your availability group role in right side pane. Right click on this role and select “Add Resources” then click on “Client Access Point“.

You will get below screen to enter listener details. I entered listener name into name section and listener IP in to Address section. Once i clicked at Next button to proceed. I got below error which says this listener is already in use in Active Directory.

As we know, listener name and IP should be unique and unused so here we need to remove this listener details from Active Directory. Contact your domain administrator and tell him to remove the entries for this listener name and IP.

Once listener details will be cleared from Active Directory, go ahead and try to add listener name to the existing AlwaysOn Availability Group. This time you would be able to create the listener name. If you are still facing some issue, then we will follow remaining steps of the above suggested MSDN article. Read next section to follow remaining steps to fix this issue.

Failed to add Listener after clearing DNS entries and DNS Status showing The handle is invalid

If you are still facing same issue during adding listener or configuring AlwaysOn Availability Group then you can again ask your domain admin to relook into Active Directory because there is a possibility of some entries in AD for the listener name and IP during your failed attempts to create the listen name. If domain admin will not find any entry then we will proceed with the steps that Microsoft has suggested in above attached article.

We will create Client Access Point in failover cluster manager for Availability Group role as we were trying to create in above section. Right click on Availability Group role and select “Add Resources” then “Client Access Point” as shown in image no 3. You will get below screen to enter listener name and corresponding IP address in highlighted sections. Click on Next button to proceed post entering the listener details.

You will get below screen to proceed once you click on next button of above screen.

Click on Finish button to create the Client Access Point in failover cluster manager for this listener.

Once you will click on finish button, your Client Access Point for your listener will be created in the failover cluster manager as shown in below screen. Availability Group role was running fine but after adding Client Access Point it has changed into Partial Running mode.

We can see CAP is showing as offline in above image. That is the reason Availability Group role has changed into Partial Running mode. Now we will bring it online manually. Right click on Listener CAP and click on “Bring Online” option.

We can see Listener IP has come online whereas listener name is failed to come online. Now we will check the properties of the client access point (CAP).

Right click on Listener Name and choose Properties option. You will get below screen where you can see NetBIOS, DNS and Kerberos status. NetBIOS and Kerberos status is showing OK but DNS status is not ok and showing “The handle is invalid.” This is the issue that needs to be fixed to create listener.

When i checked the cluster events, i saw below error details that are also very much same.

Cluster network name resource ‘Cluster Name’ failed registration of one or more associated DNS name(s) for the following reason:
The handle is invalid.
Ensure that the network adapters associated with dependent IP address resources are configured with at least one accessible DNS server.

To fix this DNS issue ( The handle is invalid ) we will repair this CAP. Right click on Listener Name showing in failover cluster manager, click on “More Actions” and then click on Repair as per below image.

Once you will click on Repair, it will come online and then go in offline. Now right click on Listener name again and click on Bring Online button. This time Listener Name will come online if CAP has been repaired successfully. You can repeat the repair exercise if your CAP is not coming online. If you are still facing issue then you can again contact to domain admin to look into Active Directory server.

Now you can see Client Access Point for the listener is showing online now. if you go to the property of this Listener CAP, you can see NetBIOS, DNS and Kerberos status is showing OK.

Close above property window. Now DNS issue is resolved that was main culprit during listener creation so next step is to remove this Client Access Point for this Listener and create the listener in SSMS that will create Listener client access point (CAP) automatically in the failover cluster manager.

Once you click on remove button below window will come to proceed. Click on ok to proceed.

Once this CAP will be removed then connect to primary replica of the AOAG configuration. Expand the AlwaysOn High Availability folder and then Availability Group Listeners folder. You will not be able to see anything as there is no listener as of now. Right click on Availability Group Listeners folder and click on “Add Listener…” option.

You will get below image (New Availability Group Listener Wizard) to enter the listener details. Enter the Listener DNS Name then port followed by choosing Network Mode. Click on Add button to enter Listener IP Address. Once you are done with all details. Click on Ok button to apply this changes. Add Listener wizard will disappear if listener will be created successfully otherwise it will give you the error on the same wizard.

As we have created Availability Group Listener for this AOAG configuration so you can go ahead and verify this in SSMS by expanding Availability Group Listeners folder. You can see same listener name is showing under Availability Group Listeners folder on both primary as well as secondary replica in below image.

Related Articles:

• Learn what is AlwaysOn Availability Group.
• Fix Listener Error 19471 due to Kerberos Status”The handle is invalid”
• How to fix AlwaysOn Availability group Automatic Failover Issue.
• Fix: Secondary Replica is showing in Resolving state and not becoming Primary
• How to fix error 15141: The server principal owns one or more endpoint(s) and cannot be dropped

I hope you like this article. Please follow us on our facebook page and on Twitter handle to get latest updates.

The post Fix SQL Server Error 19471: Listener issue that came during Configuring SQL Server AlwaysOn Availability Group appeared first on Techyaz.com.

Automatic Failover Worked Well but Application failed to connect

I have an AOAG configuration with Automatic failover between two replicas in my environment.  Last week, we had High Availability testing for all applications. Everything was working fine if we were doing failover from Management Studio. Databases were failing over and failing back successfully.

Issue occurred when application team tried to connect to database server after failover. They were failed to connect to the secondary replica after every failover but applications were working fine from primary replica. Below are the error details that are coming to application team during establishing database connection post failover to current primary replica (earlier secondary replica).

When we saw at the database level, everything was working fine. Secondary replica was successfully transitioning as primary replica after failover and databases were accessible from failover node but application was not able to connect to this replica. We checked all respective logins associated with the application and everything was there on secondary replica.

One thing we have noticed during troubleshooting that SQL login was became orphaned after each failover and we did resolved orphaned user issue manually post each failover. But there is no benefit of automatic failover if you need to establish database connection manually after each failover.

After spending some time, we fixed this issue. Application was not able to connect to database post failover because one of the SQL login which was used by application has different SID on secondary replica than its primary replica. I recreated this login on secondary replica with same SID which was in primary replica and this has fixed the issue. Here, I will explain step by step method to make you understand the issue and its solution.

Resync the SID from Primary Replica

First check the current configuration of this AOAG. Run below command to get the AOAG details. We can see the configuration is running into automatic failover mode along with replica names.

SELECT replica_server_name, availability_mode_desc, failover_mode_desc
FROM sys.availability_replicas

If you do failover between replicas then it will not throw any error in SSMS or database level. But if you will ask application team to connect to database server using Listener post failover, you will get error.

You will face error because SID of the login used by application is different on both replica servers. You can also get the error when the login used by application to establish database connection is not present on secondary replica. You need to sync the login, password and SID of this login to the secondary replica. Follow this tip to fix the issue that are coming due to any of the given reason.

The SID identifies the security context of the login and is unique within the server instance. If the login is created from a Windows user or group, it is given the Windows SID of the source principle; the Windows SID is unique within the domain. If the login is created as a legacy-style SQL Server login that requires a password, the server will generate a SID. Now check the SID of the login which is used to connect to database on primary replica.

--Replace your identified login with xyz.
select name,sid from syslogins
where name='xyz'

Now run same command on secondary replica. You can see SID is different on both replicas for same login. Ideally everything should be same and in sync with primary replica that are associated with AOAG.

Now we need to delete this login from secondary replica and recreate it using same SID which is present on primary replica. Next, drop the login either using SSMS or T-SQL. SSMS way is given below:

• Connect to target SQL Server Instance.
• Expand the security folder.
• Expand the Logins folder.
• Right click and choose delete on the identified login which needs to be deleted.
• Click on Ok button of the login deletion window.

If you want to drop this login using T-SQL command then open a query window and execute below command.

--change the name of xyz with your login name which you want to delete.
DROP Login 'xyz'

You might get multiple dependency errors during dropping this login because there is possibility, this login might own few objects, agent jobs or databases on this instance. I got Error 15170 during dropping this login which i fixed in attached tip to remove this login. You can also read below articles if you are getting different kind of DROP Login failed issues:

• How to fix Error 15174: Login owns one or more databases.
• How to fix Error 15173: Revoke the permissions before dropping the login.
• How to fix error 15138 & Error 3729: DROP login failed
• How to fix error 15141: The server principal owns one or more endpoint(s) and cannot be dropped
• How to fix error 15434: Could not drop login as the user is currently logged in

Once login will be dropped, go ahead and create same login with same password and SID which are used on primary replica. Copy the SID from image 2 when you check the SID for this login on primary replica and use that during CREATE Login statement. If you don’t have this login on secondary replica yet then also you can follow same process to create the login.

--copy the SID from primary replica.
CREATE Login xyz WITH password = 'password@123', CHECK_Policy=Off, SID = 0xC2Cabcdefghijk123hghj6C3

We can see command executed successfully. Now issue will be fixed if you don’t had this login on secondary replica and you just created it for very first time. But you might get orphaned user issue if you had this login earlier with different SID on this instance. Now we need to fix the orphaned users on the secondary replica.

Fix Orphaned Users

Now you need to run below command to check the orphaned users on your database. Make sure that your database is working as primary otherwise you would not be able to run this command. You can failover and then run this command if database will become primary.

--change the dbname
USE DBNAME
Go
sp_change_users_login 'report'

If you will get your user name which was mapped to the identified login then we need to fix this orphaned user. Run below command to fix this issue.

--change the dbname
--Map database user xyz to login xyz.  
USE DBName;  
GO  
EXEC sp_change_users_login 'Update_One', 'xyz', 'xyz';  
GO  

Now orphaned user issue is also resolved and we have fixed the application connectivity issue that was reported after automatic failover of AOAG. You can do failover testing again to make sure application connectivity is working fine after AOAG failover.

I hope you like this article. Please follow us on our facebook page and on Twitter handle to get latest updates.

The post Fixing SAP Connectivity issue after AOAG Automatic failover appeared first on Techyaz.com.

SQL Server AlwaysOn Availability Group

SQL Server AlwaysOn Availability Group is an advanced version of database mirroring and introduced in SQL Server 2012. AOAG is HA & DR solution for SQL Server databases. We can group a bunch of databases as one Availability Group and failover/failback as one entity whereas it was not possible in database mirroring. Database Mirroring can be configured only for single database at a time. You can visit the attached link to learn more about AlwaysOn Basics.

I have also an AOAG configuration in my environment which is set for Automatic failover between both replicas. Last week, we faced an issue during automatic failover. Secondary replica was not transitioning as primary replica during an automatic failover and it was stuck in “Resolving” state. It was full down time for application because primary replica was lost and availability databases were not coming online on secondary replica.

I will explain step by step method to make you understand the issue and the fix.

Automatic Failover Not Working and Secondary Replica Stuck in Resolving State

As secondary replica is not transitioning as primary replica after failover because secondary replica hung in resolving state so here, we will check the main root cause and its fix so that secondary replica comes online after every failover. Let’s check the current configuration of this AOAG. Run below command to get the AOAG details. We can see the configuration is running into automatic failover mode.

SELECT replica_server_name, availability_mode, availability_mode_desc, failover_mode, failover_mode_desc
FROM sys.availability_replicas

Now connect to secondary replica in SQL Server Management Studio. Expand the AlwaysOn High Availability folder followed by Availability Group folder. Here, you can see the current state of this availability group which is showing in Resolving state.

I checked the SQL Server error log file but i did not get enough information about this issue to proceed. I searched about this issue and got few suggestions on web that i should check the cluster log. I checked failover cluster manager for critical events. I got below event in failover cluster manager.

Clustered role 'AG_***' has exceeded its failover threshold. It has exhausted the configured number of failover attempts within the failover period of time allotted to it and will be left in a failed state.
No additional attempts will be made to bring the role online or fail it over to another node in the cluster.
Please check the events associated with the failure. After the issues causing the failure are resolved the role can be brought online manually or the cluster may attempt to bring it online again after the restart delay period.

Fix Resolving state and Make Secondary Replica Primary

Above issue clearly indicates about some cluster configuration related to role created for AOAG. As we can see, there is one role is showing under roles tab of this cluster group in below image. We can see the current owner of this role is still showing as node 1.

Now right click on this role and choose properties as per below image.

You will get a properties window as shown in below screenshot. Now click on “Failover” tab of this screen to check the failover threshold details.

We can see the failover threshold is set to only 1 in the 6 hours. It means if any failover happen more than once in the 6 hrs time period it will remain in failed state and will not try to come online. Now increase this value as per your need. I have changed it to 10. Click on apply and Ok button.

Now, secondary replica of availability databases will not come online automatically. We need to bring them online. We have multiple options to bring them online from resolving state. You can directly run the failover by right click on availability group in SSMS and then proceed with failover.

Another option is to bring them online from failover cluster manager. You can either right click on the AOAG role and click on “Bring Online…” option or you can failover it to best possible node. As primary replica is down and not accessible so it will be failed over to the secondary replica only because this node is only available this time. Let’s right click on AOAG role in failover cluster manager, choose move and then click on “Best Possible Node”

It will process and come online within few seconds and this time you can see this role is online from node 2 as current owner is showing as node 2 now.

We can also validate this change while making a database connection to the secondary replica in SSMS. Connect to secondary replica and expand the AlwaysOn Availability Group folders. You can see now availability group is showing as primary. That means your database is online from secondary replica to accept the user connections.

You can also test this exercise to make sure automatic failover is working fine or not to prevent this kind of future outages. Once your earlier primary replica comes online, you shutdown SQL Server services on secondary replica/current primary replica. Once it will go down, your current secondary replica becomes primary automatically. Before shutdown SQL Server services, make sure databases should be fully synchronized.

Hee, we have fixed Alwayson issue where secondary replica is not coming online after failover and hung in resolving state. I hope you like this article. Please follow us on our facebook page and on Twitter handle to get latest updates.

Read More:

• How to Upgrade or Apply Patches on AOAG Instances?
• Fix AOAG Listener Error 19471: The handle is Invalid
• How to Restore a Database participating in AOAG?
• Fix Application Connectivity Issue after AOAG Failover

The post Why is Secondary Replica Showing in Resolving state after AOAG Automatic failover? appeared first on Techyaz.com.