We had few options to overcome this issue. As we were running this database in simple recovery mode, so we didn’t have an option to run the Transaction Log backup. We tried to shrink the log file to manage transaction log file but it was not allowing us to do so. This might be due to the data load execution was in process and active transaction.

Finally, we left out with two options; either extend the log drive or add a log file to the database on a drive that has enough space to accumulate the logs. Extending the existing drive was not possible because data load was in progress and we need to fix this issue without stopping the import/data load. So, we decided to go with another option that was to add a database log file to a drive that has enough space. I have given you step by step method to add a log file to a database in below section.

Add Log File to a Database

I will show you how to add log file to a database using T-SQL statements and using GUI in SQL Server Management Studio to manage transaction log file during data load. This way you can manage if your transaction log file is growing faster.

Using T-SQL Statement

Open new query window and run below T-SQL statement to add a log file TY1_Log3.ldf to database TY1 on F drive. Have a look at below T-SQL code to get other details like sizing and growth details.

--Replace your Database Name and file details in below T-SQL Code.
ALTER DATABASE [TY1]
ADD LOG FILE (NAME = 'TY1_Log3', FILENAME = 'F:\TY1_Log3.ldf', SIZE = 20080 MB, FILEGROWTH = 5024 MB)
GO

Once you execute above T-SQL statement, your log file TY1_Log3.ldf will be added on F drive to the database TY1.

Remember, add the log file in a drive where you have enough space. If you will add log file to data file drive then you might face IO pressure because both data and log files will be on same drive so avoid to keep data and log file on same drive.

Using GUI in SSMS

1. Launch SQL Server Management Studio and connect to the instance of the SQL Server Database Engine. Expand Databases folder, right-click the database in which you have to add the log file, and then click Properties.
2. In the Database Properties dialog box, select the Files page from left side pane.
3. To add a transaction log file, click Add button from lower right side pane.
4. In the Database files grid, enter a logical name for the file. The file name must be unique within the database. Select the file type as log. Specify the initial size of the file. Set Autogrowth column as per your requirement.
5. Specify the path for the file location along with physical log file name. The specified path must exist before adding the file. Make sure that your drive has enough space to host this log file.
6. Click on OK button to create this add this log file to the database.

Once you performed step 6, log file will be added to your database on your identified drive name. These exercises (adding log file to database) doesn’t require any downtime or it will not interrupt any existing transactions. This way you can manage transaction log file during data load if your transaction log file is growing faster.

Read More

• How to get Total no. of Virtual Log Files in a Transaction Log file?
• Understanding Transaction Log File Architecture
• Why Should You Always Turn Off Database Auto Shrink Property?
• What is the best value for database file autogrowth?

I hope you like this article. Please follow our Facebook page and Twitter handle to get latest updates.

The post Manage Transaction Log File during Data Load appeared first on Techyaz.com.

Let me first give you some background about SAP migration tool like how it works. SAP Migration tool first captures every related information along with data and creates an export file that needs to be copied to the target location or server and then we perform bulk data import with the help of SAP migration tool. Bulk data import or data load will be done locally and not over network.

When we ran SAP migration tools to capture all details, a dump file was created that we have copied to the SQL Server box. We had approx. 900GB of data that needs to be loaded into SQL Server database. When we performed SQL Server data load using SAP migration tools with standard settings, it took around 14 hrs. to complete the bulk data import.  As 14 hrs. was very long time so we decided to work on reducing the total time taken by import process.

We made some changes from both sides application as well as SQL Server database side. SQL Server data load time has reduced drastically to around little more than 6 hours post these change implementations. Here, I am explaining each parameter that we have configured from database side just for this bulk data import.

NOTE: Make sure to revert these changes as per your application/vendor best practices post this bulk data import for OLTP/normal operations. We are changing value considering Data Load only.

It is important to understand the factors that affect bulk data import performance. To speed up the data load, you need to look into below configuration options from DBA standpoint.

Auto Grow

 Autogrow settings can be used to automate SQL Server database file growth. This is very useful property but this can degrade your bulk import performance or data load operations. Read attached article to understand SQL Server Autogrow setting in detail.

Autogrow setting increment must be large enough to avoid the performance penalties.  If you set default or small value during bulk import or data load, database engine will have to perform data file autogrow operation every time the file will be filled with data till maximum size limit. More autogrow operation will take more resources like IO, CPU etc and time to complete the data load transaction.

The exact value for this setting should be based on data growth. You should not be left autogrow in percentage value rather you should use some value in MBs. A general rule of thumb is to set your autogrow setting to about one-eighth the size of the file. Please read attached article to understand how to manage transaction log files during huge data load.

I would suggest keeping database size as much as you are expecting your database will grow to avoid autogrow operation. If you are aware that your database size will be 900 GB post data load, you should set your database size to 900 GB before starting the data load. This will improve the bulk import performance. Make sure to enable IFI (Instant File Initialization) for your SQL Server instance to reduce total time SQL Server takes to set the database size.

Instant File Initialization

As I suggested in above point, Turn IFI (Instant File Initialization) ON to speed up AutoGrow process. If you set a considerable large value to autogrow settings, it will take some time to increase the size by zeroing the disk address. IFI speeds up this process and allow us to create enormous size of data files in quick way. Read attached article to learn more about IFI (Instant File Initialization) and how it will speed up autogrow or data load operation. You will also learn how to enable IFI for your SQL Server instance in this attached article.

Auto Shrink

AutoShrink is very costly operation that should be always OFF even during your daily and normal operations. Turn it OFF immediately. Make sure your disk has enough space to sustain during bulk operations or data load. Repeatedly shrinking and growing the data files will cause file-system level fragmentation, which can slow down performance. It also takes lot of CPU and IO. Read attached article to know why you should always keep Auto_Shrink off.

Reduce Logging

Change recovery model to SIMPLE or BULK-LOGGED to reduce the logging operations. The simple recovery model minimally logs most bulk operations. For a database under the full recovery model, all row-insert operations that are performed during bulk import are fully logged in the transaction log.

For large data imports, this can cause the transaction log to fill rapidly. For bulk-import operations, minimal logging is more efficient than full logging and reduces the possibility that a bulk-import operation will fill the log space. To minimally log a bulk-import operation on a database that normally uses the full recovery model, you can first switch the database to the bulk-logged recovery model. After bulk importing the data, switch the recovery model back to the full recovery model.

Auto_Update_Statistics

Auto_Update_Statistics is very useful property that should be set to ON during normal business operations but here I would suggest turning it off during huge data load or bulk import operations. Updating the statistics also takes resources from the system. This can lead to performance issues when you are doing bulk data. It’s important to keep up-to-date statistics for the optimizer to use, but here we will turn it off and once data load or bulk data inserts will be performed, we will again enable it.

Auto_Create_Statistics

This should be OFF because creating statistics also takes resources from the system that can degrade the overall bulk import performance. Make sure to enable it post data load or bulk inserts operation.

Parallelism – MAXDOP

SAP suggests setting MAXDOP value to 1 during normal business operation so we had this value during first data load that took around 14 hrs. We had assigned half of the CPU cores for the parallel processes during data load and bulk import operations. I believe MAXDOP has contributed significant role in reducing total data load time. Make sure to revert this change to its original value as per your application best practices.

You can change MAXDOP settings either using sp_configure command or using GUI in SSMS by launching SQL Server Instance property window.

Using Batches

Make sure to use an optimum batch size that will increase the bulk data import performance. Importing a large set of data as a single batch can be problematic, so bcp and BULK INSERT let you import data in a series of batches, each of which is smaller than the data file. Each batch is imported and logged in a separate transaction, and after a given transaction is committed, the rows imported by that transaction are committed. If the operation fails, only rows imported from the current batch are rolled back, and you can resume importing data starting at the beginning of the failed batch rather than at the beginning of the data file.

I hope you like this article. Please follow us our Facebook page and Twitter handle to get latest updates.

The post How to Improve SQL Server Bulk Data Import Performance? appeared first on Techyaz.com.