The SSISDB database (a.k.a. the Integration Services catalog) was introduced back in SQL Server 2012 as a way to de-clutter MSDB database and provide some built-in logging and reporting infrastructure. In a nutshell, SSISDB is an SSIS framework (see Fig #1) making SQL Server Integration Services more robust and enterprise-friendly by providing the following features:

• Database backup
• Database encryption
• Support for environments
• Support project and package parameters
• Package versioning
• Custom SSRS performance reports built into SSMS
• Support for deployment within SSDT

While overall SSISDB Catalog was and still is a great enhancement to SQL Server Integration Services, some previously working fine feature took a wrong turn. This blog post is going to cover the “behavioral” changes that Microsoft made to SQL Server Agent job notification as part of the SSISDB Catalog, specifically the error message reporting on failed package.

Problem

Prior to SQL Server 2012, an email notification on a failed SQL Server Agent job step with an SSIS package was really informative. The error message was specific enough, as it was providing the error from the SSIS execution engine. When Microsoft introduced SQL Server 2012, it decided to replace this specific SQL Server Agent job email notification with a generic message directing the user to the SSIS Catalog error log (see Fig #2).

I’m assuming when most of us are receiving Operational emails, we want to know not only if to react or not, but also how to react. This is especially important now, since we use mobile devices to read emails and might not have access to the a real computer.

Unfortunately, the email above gives only an indication that some SSIS package failed without giving any details. For example, in the past (prior to SQL Server 2012), you would know that your package failed due to connectivity issue by just reading an email and decide to ignore the notification. Now you don’t have that option. You would need a real computer to fire-up SSMS and run a custom SSRS on that package, adding many clicks to see the details error. Needless to say, running to your desktop/laptop to open SSMS only to discover connectivity problems is unnecessary overkill.

Solution: Use T-SQL to create a better SQL Server Agent job email notification 

The solution will involve a T-SQL query (I should probably make it a stored procedure at some point) that would connect tables and records between both MSDB and SSISDB databases to bring back a detailed message on a failed SSIS package. Equipped with an email that has detailed error message, you can react without leaving your couch – fix it or just let it retry.

In a nutshell this solution is to going to:

1. Find the job name based on a current SPID
2. Find a job with a failed step
3. Connect step with SSIS package name
4. Use the SSIS package name to find run-time errors in SSISDB catalog
5. Bubble us to an inner exception to include only detailed error
6. Locate a default job operator
7. Send a detailed error report

The following script would need to be added as last step in a job (you can disable the job alert) with all the steps that include SSIS packages going to that last step on failure (Fig 3).

Additionally, the used-to-be last step before Report SSIS failure was introduced that will now precede that step will need to quit the job reporting success (Fig 4) instead of continuing to the next step..

Source code

DECLARE @Full_ssis_command VARCHAR(4000),
 @Job_step_id INT,
 @Package_name VARCHAR(4000),
 @EMailBody NVARCHAR(2000),
 @EMailSubject VARCHAR(150),
 --@Job_name VARCHAR(100),
 @Job_id UNIQUEIDENTIFIER
 
SELECT	@Job_id = Job_id
		--@Job_name = [name] --AS job_name
FROM	sys.dm_exec_sessions AS ions
	INNER join msdb.dbo.sysjobs AS jobs ON jobs.job_id
				= CAST(CONVERT(BINARY(16), SUBSTRING(ions.[program_name],CHARINDEX('(Job 0x',ions.[program_name],1) + 5 , 34),1) AS UNIQUEIDENTIFIER)
WHERE	1=1
	AND ions.session_id = @@SPID --63
 
--print @Job_name
 
SELECT TOP 1 @Job_step_id = Step_id
FROM msdb.dbo.sysjobhistory (NOLOCK)
WHERE Run_status <> 1
 AND Step_id > 0
 AND Job_id = @Job_id
ORDER BY Instance_id DESC
 
SELECT @Full_ssis_command = Command
FROM msdb.dbo.sysjobsteps (NOLOCK)
WHERE Job_id = @Job_id
 AND Step_id = @Job_step_id
 
 PRINT @Full_ssis_command
 
IF @Full_ssis_command LIKE '%.dtsx%'
 BEGIN
 SELECT @Package_name = RIGHT(LEFT(@Full_ssis_command,CHARINDEX('.dtsx', @Full_ssis_command)), CHARINDEX('\', Reverse(LEFT(@Full_ssis_command, Charindex('.dtsx', @Full_ssis_command)-1)))) + 'dtsx'
 END
 
SELECT TOP 1 @EMailBody = LEFT([Message], 2000), @EMailSubject = 'Package : ' + Package_name + ' failed ON :' + CONVERT(VARCHAR,Message_time)
--[Message_time],
-- [Extended_info_id],
-- [Package_name],
-- [Message_source_name],
-- [Subcomponent_name],
-- [Package_path],
-- [Execution_path],
-- LEFT([Message], 400)
FROM SSISDB.[catalog].[Event_messages] (NOLOCK)
WHERE [Package_name] = @Package_name
 AND Event_name = 'OnError'
 AND Message_time >= DATEADD(DD,0,DATEDIFF(DD,0,GETDATE()))
 AND Operation_id IN (SELECT Max(Operation_id)
 FROM SSISDB.[catalog].[Event_messages](NOLOCK)
 WHERE [Package_name] = @Package_name)
ORDER BY Message_time ASC
 
DECLARE @operator_name AS VARCHAR(100) = (SELECT top 1 name from msdb.dbo.sysoperators)
PRINT @operator_name
 
exec msdb.dbo.sp_notify_operator  @profile_name = NULL ,
    @id = NULL,
    @name = @operator_name,  
    @subject = @EMailSubject,  
    @body = @EMailBody

Here is the improved SQL Server Agent job email notification with a detailed message that we can expect now (Fig 5).

The post Bringing back an informative SQL Server Agent job email notification on failed SSIS packages appeared first on SQL Solutions Group.

Introduction

The SSISDB database (a.k.a. the Integration Services catalog) was introduced back in SQL Server 2012 as a way to de-clutter the MSDB database and provide some built-in logging and reporting infrastructure. In a nutshell, SSISDB is an SSIS framework making SQL Server Integration Services more robust and enterprise-friendly by providing the following features:

• Database backup
• Database encryption
• Support for environments
• Support project and package parameters
• Package versioning
• Custom SSRS performance reports built into SSMS
• Deployment directly from SSDT

Since SSISDB Catalog is going to log package run-time information, Microsoft created a SQL Server Agent job (SSIS Server Maintenance Job, see Fig .1) to do the in-house cleaning. This job is running two steps (see Fig 2) to enforce the data retention policy (based on SSISDB Catalog settings) instead of letting all the data collection activities to eat all your disk space and effectively brake your SSIS run-time service (if logging is enabled). 

While SSIS Server Operation Records Maintenance (a.k.a. step #1) is in charge of removing all the old (default=365 days) logging, SSIS Server Max Version Per Project Maintenance (a.k.a. step #2) is in change of removing all old (default=10 versions) package versioning.

Problem

“SSIS Server Maintenance Job” SQL Server Agent job started to fail  with the following error:

Executed as user: #MS_SSISServerCleanupJobLogin##. A cursor with the name ‘execution_cursor’ does not exist. [SQLSTATE 34000] (Error 16916)

This run-time error (trying to deallocate a CURSOR that doesn’t exist) happens inside Step #1 which runs a ssisdb.internal.cleanup_server_retention_window stored procedure. I’m  going to speculate and say that this bug was unintentionally introduced by adding more features (most likely, it’s the server_operation_encryption_level which was introduced with SQL Server 2012 SP2). While I first spotted this problem in SQL Server 2017 with CU12, unfortunately it’s still a problem as of December 2019 with the latest and greatest CU18 (14.0.3257.3).

Solution

You can wait for Microsoft to fix it in a next CU (a.k.a. Cumulative Update) or you can fix it yourself by modifying the code inside ssisdb.internal.cleanup_server_retention_window stored procedure.

A little warning here: You are going to be modifying the code that was created by Microsoft. You need to be comfortable with taking that risk.

You would need to alter the procedure by moving a DEALLOCATE statement from line #175 to #177 to fix this failed SSIS job (see Fig #3).

This post is based on this MSDN post.

The post How to fix a failed “SSIS Server Maintenance Job” job appeared first on SQL Solutions Group.

This makes perfect sense to Visual Studio developers, as extensions are the normal way for extending the platform to do different things. But for DBAs and BI developers who may not be accustomed to this, it can be a bit confusing.

I won’t re-invent the wheel here by showing how to install each of these, as there are plenty of resources out there. The basics are that the database projects (i.e SQL Server Data Tools) are installed as a Visual Studio Workload, and the SSIS, SSRS, and SSAS are installed as extensions (although the SSIS extension comes as an executable rather than a .vsix file).

Who on your team is proactively protecting your

SQL Server estate? If your strategy is rubbing a

rabbit’s foot and keeping your fingers crossed,

let’s talk before something goes wrong. 

Because it will.

However, one of the things I discovered is that even after adding the Integration Services extension, it is not very intuitive how to create a new SSIS project. In the past, we had the Business Intelligence Category of project and would just choose it from there. But in Visual Studio 2019, when you go to create a new project, that option does not exist.

I thought it might be under Extensions, which would make sense because these are installed as extension, but that is not the case either. In fact, nothing shows up when choosing that project type.

What you need to do is enter Integration Services or SSIS in the search box to find it.

Once you’ve created your Integration Services Project, you’ll see the familiar project layout from prior versions.

The other issue is that some of these things are hard to find.  Greg Low does a great job in this article on that one.

The post Visual Studio 2019 SSIS Projects appeared first on SQL Solutions Group.

The SSISDB database (a.k.a. the SQL Server Integration Services catalog) was introduced back in SQL Server 2012 as a way to de-clutter the MSDB database and provide some built-in logging and reporting infrastructure. In a nutshell, SSISDB is an SSIS framework making SQL Server Integration Services more robust and enterprise-friendly by providing the following features:

• Database backup
• Database encryption
• Support for environments
• Support project and package parameters
• Package versioning
• Custom SSRS performance reports built into SSMS
• Deployment directly from SSDT

Fortunately or unfortunately, installation of a new SQL Server instance does not include the creation of the SSISDB Catalog, even if SQL Server Integration Services is included in the installation. For some reason, Microsoft decided to leave that part to the user (luckily, this is a one-time sunk cost).

Once SQL Server installation is completed, you would need to create Catalog using SSMS (see Fig #1) by expanding Management, right clicking on Integration Services Catalogs, and clicking on Create Catalog menu item (see Figure 1).

Problem

Once you are done with the full SQL Server installation and ready to create Catalog (SSISDB Catalog to be exact), you might be “greeted” with the following error – “The catalog backup file 'C:\Program Files\Microsoft SQL Server\140\DTS\Binn\SSISDBBackup.bak' could not be accessed. Make sure the database file exists, and the SQL Server service account is able to access it.(Microsoft.SqlServer.IntegrationServices.Common.ObjectModel)“.

I’m sure you were surprised as I am! What? Why would creation of SSISDB Catalog have anything to do with SSISDB database backup (see Fig #2)? Apparently it’s one of the internal steps that Microsoft does behind the scenes when you are installing SQL Server. To be absolutely specific, installation of an Integration Services feature. Strange, right? The real problem is that when you are trying to create an SSISDB Catalog, Microsoft “assumes” that there is database backup file (see Figure 2).

Solution

Apparently, if you forget to enable installation of Integration Services feature, this is the message that you are going to see. The solution involves going back to the installation (see Figure 3) and checking the Integration Services feature.

Once you have Integration Services enabled (see Figure 4) and the installation completed, you should be able to Create Catalog without any problem.

Disclaimer

This blog post is based on the following forum discussion https://www.sqlservercentral.com/forums/topic/missing-ssisdbbackup-bak.

The post Can not create Catalog (SSISDB) due to an error appeared first on SQL Solutions Group.

Introduction

The SSISDB database (a.k.a. the Integration Services catalog) was introduced back in SQL Server 2012 to de-clutter the MSDB database and provide an in-house logging and reporting infrastructure. In a nutshell, SSISDB is an SSIS framework making SQL Server Integration Services more robust and enterprise-friendly by providing:

• Database backup
• Database encryption
• Support for environments
• Support project and package parameters
• Package versioning
• Customer SSRS performance reports built into SSMS
• Deployment directly from SSDT

While SSISDB framework made SSIS much more capable, it came with some Microsoft assumptions (a.k.a. defaults). Those defaults are there to help you to stand on your feet, but might not be optimal when you start running, and are far from perfect when you are sprinting.

This article addresses the defaults, why those defaults might not be optimal, and how to change those defaults.

Problem

SSISDB Catalog ships with a built-in process to clean up operations and project versioning. This cleanup process relies on SSISDB defaults that might make your SSIS inoperable if not changed.

Virtually every new SSIS project starts with building and running just a few packages while there is an abundance of free disk space. Fast forward 3-6 months and you have a number of SSIS projects, tens or hundreds of packages, and you might also have a need for a fresh data for reporting and analytics (read: constantly running packages around the clock). Those SSIS packages might accumulate versioning data (less of a problem) and logging data (more of a problem) way beyond Microsoft cleanup defaults which might lead to SSISDB database size to balloon and grow in size unexpectedly (read: eat up all the available disk space).

The single default that is responsible to capture SSIS run-time related data is Server-wide Default Logging Level = Basic. This setting is ultimately what manages the SSISDB database size and ultimately taking up the disk space. That being said, SSIS Catalog (Fig 1) provides other options to control the SSISDB database size/disk space:

1. Server-wide Default Logging Level
2. Clean-up defaults:
   1. Clean Logs Periodically = True
   2. Periodically Remove Old Versions = True
   3. Retention Period (days) = 365
   4. Maximum Number of Versions per Projects = 10

While changes to Server-wide Default Logging Level from Basic to None to regain disk space is highly unpractical and will make troubleshooting a nightmare, changes to other SSISDB Catalog settings like Maximum Number of Versions per Projects and Retention Period (days) can have minimal effect on troubleshooting, while keeping SSISDB database/disk space in control.

All those defaults are driving the cleanup process (a.k.a. SSIS Server Maintenance Job). While both Clean Logs Periodically = True and Periodically Remove Old Versions = True are virtually always perfect, settings for both Maximum Number of Versions per Projects = 10 and Retention Period (days) = 365 might be overoptimistic. And here is why.

The main reason why both Clean Logs Periodically = True and Periodically Remove Old Versions = True defaults are perfect is, without those defaults (or effectively if both changed to False), SSISDB will grow indefinitely as it will only accumulate log and versioning data without removing any data and run out of disk space.

The main reason why both Maximum Number of Versions per Projects = 10 and Retaining logging data for 365 days might be overoptimistic is Microsoft’s decision to retain the data based on a time period and not size. That means the fixed period number is not taking into account how much data will be accumulated. While Maximum Number of Versions per Projects might grow into a problem once you have thousands of packages, Retention Period (days) is going to become a huge problem once you have packages that are running every few minutes. I don’t think Microsoft ever expected accumulation of log data for 1440 runs a day (running every minute) which will require a cleanup more often than every 365 days or it will run out disk space.

Solution

The solution would involve changing the defaults to smaller numbers, so SSISDB keeps less information. Based on my personal experience of running packages every minute (e-commerce platform requirement) and having up to 10 projects, decreasing Retention Period (days) from 365 (default) to 7 days and Maximum Number of Versions per Projects from 10 (default) to 5 would suffice to control SSISDB/disk space while keeping enough information for debugging and troubleshooting purposes.

You have two options to make this change:

• Change defaults using SQL Server Management Studio (a.k.a. “SSMS”) by expanding Integration Services, right-clicking SSISDB and then clicking on the Properties menu-item (Fig 2 – SSMS).

• Change defaults using T-SQL below to update records in  ssisdb.catalog.catalog_properties table directly.

Source code

UPDATE ssisdb.CATALOG.catalog_properties
SET property_value = 5
WHERE property_name = 'MAX_PROJECT_VERSIONS'
 
UPDATE ssisdb.CATALOG.catalog_properties
SET property_value = 7
WHERE property_name = 'RETENTION_WINDOW'

Once the change is completed, you should be able to confirm that changes by running the T-SQL query below (Fig 3 – SSISDB Catalog)

Source code

SELECT * 
FROM ssisdb.CATALOG.catalog_properties

While changing defaults might fix the problem going forward, it might not fix your existing SSISDB disk space problem quick enough. You can speed up the cleanup process by modifying the code and changing @delete_batch_size from 1000 to like 10000 by altering ssisdb.internal.cleanup_server_retention_window stored procedure (Fig 4 – Retention Procedure).

Summary

This article highlighted a potential problem with SSISDB Catalog defaults that might lead to running out of disk space and provided a solution that would force SSISDB to retain less information.

The post SSISDB Catalog Defaults Best Practices appeared first on SQL Solutions Group.

Introduction

To deploy SSIS packages from SSDT to SSISDB, using Integration Services Deployment Wizard is one of the simplest and interactive ways. All you have to do is to right-click the package and click on Deploy, go through a few menus and you are done.

Problem

Unfortunately, SSIS is too bound to an Active Directory and would only utilize Windows authentication to understand “who can do what”. If both machines (source/deployed from and target/deployed to) are on the same domain or have domain trust/forest, you are in luck, but what if they are not? Additionally, if your development machine is running Windows 10 Home or iMac, you might not even have an option of the domain.

Additionally, we are in a public cloud era, so your company can have private and public cloud domains that are not connected (trusted). That makes the deployment from one machine to another quite a pain.

Solution

What if you could type the AD credentials of the machine you are trying to deploy to when firing up SSDT on your development machine? In that case, Integration Services Deployment Wizard will see the proper credentials and everything will work like it was on the same domain.

Sysinternals comes to rescue – https://docs.microsoft.com/en-us/sysinternals/downloads/shellrunas. ShellRunAs allows you to specify a different username/password when opening a new application. It is very similar to RunAs Administrator except that you can type a new username and password. Once SSDT gets the credentials of the target machine, everything works like a charm.

Thanks go to Randy Knight for showing me that trick to deploy SSIS packages! I wish Microsoft would incorporate that handy tool into SSDT to make deployment more consistent.

If you still want to take a step back and create a new SSIS package, we recommend taking a look at the guide prepared by SentryOne. You can find information on how to create an SSIS project to add a data flow task.

The post How to deploy SSIS packages using SSDT from an untrusted domain appeared first on SQL Solutions Group.

Today I want to share a couple of examples.  We will take a look at examples of both good and bad implementations of synonyms.

First, let’s take a look at the syntax for creating a synonym.  Per BOL (and for the fun of it, here is the 2005 link).

Source code

-- SQL Server Syntax
 
CREATE SYNONYM [ schema_name_1. ] synonym_name FOR <object>
 
<object> :: =
{
    [ server_name.[ database_name ] . [ schema_name_2 ]. OBJECT_NAME 
  |  database_name . [ schema_name_2 ].| schema_name_2. ] OBJECT_NAME
}
-- Windows Azure SQL Database Syntax
 
CREATE SYNONYM [ schema_name_1. ] synonym_name FOR < OBJECT >
 
< OBJECT > :: =
{
    [database_name. [ schema_name_2 ].| schema_name_2. ] OBJECT_NAME
}

So a sample implementation of a Synonym could be something like the following.

Source code

CREATE SYNONYM [syn].[Staging_Cardumum] FOR [Cardamum_UAT].[dbo].[cardamum_staging]
GO

The OLE DB provider “SQLNCLI10” for linked server “blahblahblah” indicates that either the object has no columns or the current user does not have permissions on that object.

The error message seems at least semi-descriptive and gives a starting point.  In this case, I decided to verify the linked server was created properly, verified that the permissions were done properly and even tested the linked server.  On the source (linked) and destination server (let’s call the Source server ServerA and the Destination server we will call ServerB), I verified that permissions were in place for each database to be touched.  Still no dice!

Well, let’s go take a look and see if that referenced table actually exists.  It did not!  Does it exist as a view?  It did not!  Alas, the table existed as a synonym.  This is where it gets wonky.  In looking at the definition of the synonym, I found that the table defined in the synonym had a linked server table as its source.  To top things off, the linked server was back on the originating server that was coming across the link in the first place.  So yes, that would be ServerB initiated a query against ServerA to pull data back to ServerB.  But the data needed (as defined by the vendor) was available on ServerA – supposedly.  Reality had that data actually sitting on ServerB the whole time.

At any rate, thanks to having a synonym for each and every table sitting on ServerA that referenced a table across a linked server on ServerB, we had mass confusion.  In the end, the query was far simpler to execute by just pulling it from the originating query server (ServerB).

This implementation of a synonym was not the best.  All it did was cause confusion, create documentation inaccuracies and delay the developer from accomplishing her task.  Do you really need 1000s of synonyms in your database?  Do you need all of them going across a linked server?  If you do, did you ever think about the potential for performance issues?  (The vendor in this case used those linked servers and synonyms to perform a data conversion that took 36 hrs each time for a rather small dataset – ouch!!!!).

On the other Hand

Imagine, if you will, two databases sitting on the same box.  One database will be for your user data, and the other for “staging” data for processes such as those related to ETL.  Imagine further that, for some inane reason, the naming standard of your databases must include the suffix denoting the environment of the database.

Now picture an SSIS package that must utilize tables from both databases in data sources somehow.  At least one of the tables has millions of records.  And both of the tables have nearly 100 columns.  Even just pulling in the minimum amount of data using a query from each source can cause memory issues.  Two separate data sources means you will likely have a sort transformation (for each source) as well as a join transformation.

Trying to reduce the amount of data in the larger table source could be done via TSQL.  But to reference a database in one environment versus another environment means a code change with each deployment (due to the TSQL – think three part naming).  So you have been hampered by the environment.  Or have you?

By using a synonym in this situation, the data can be joined in a tsql data source by referencing that synonym.  Let’s look back at the sample synonym posted earlier in this article.

Source code

CREATE SYNONYM [syn].[Staging_Cardumum] FOR [Cardamum_UAT].[dbo].[cardamum_staging]
GO

You can see that this synonym follows the same sort of naming standards as was just laid out in the preceding scenario.  If I create a synonym in each environment by the same name, and referencing the appropriate environment named database, I have just opened up a performance tweak for my SSIS datasource.

By implementing this slight tweak, I have been able to gain a 10x performance improvement in package performance.  I am now requiring SSIS to ingest fewer records and thus chew up less memory.  Fewer transformations are required and the package can just fly into the required transformations, rather than tinkering around with the transformations needed to just get the data into a usable state for those transformations.

There are other benefits within SSIS to using synonyms for databases on the same server as well.  Especially when dealing with this kind of naming standard that requires the databases to be named differently in each environment.

Conclusion

How you use a synonym can be a huge asset or it can be a significant dampener to performance.  There are benefits and uses for these nifty little things.  Check them out and let us know how you have been able to put synonyms to use to benefit you.

The post Synonyms in SQL Server – Good and Bad appeared first on SQL Solutions Group.

The fact of the matter is that computer systems have always had the GIGO (garbage in garbage out) problem. Assuming that data will come in clean is always a recipe for failure.  Instead, one must assume the data will come in dirty in a variety of ways (some yet unknown) and handle this bad data in the SSIS package.

The Problem

SSIS does a great job of error logging and providing built-in methods to redirect data to different flows based on many different configurable items. Pretty much anything you can think of to look at and decide where to send it, you can do. What it doesn’t do a very good job of out of the box is providing a generic place to put the bad data in an easily human- readable format for troubleshooting after the fact.

Observe the following common data flow error handling pattern for loading a fact table. A lookup component retrieves the foreign key from a dimension table and sends the row on through the data flow if there is a match. If it doesn’t match, the row is sent elsewhere, in this case to a text file connection. A DBA or analyst then has the responsibility to monitor the text files, find where the error was, fix the issue and reprocess the data. This is what the developer above was spending most of that time on.

There are several problems with this approach:

• I have to have a separate text file for every place in my ETL process where I might encounter bad data.
• All that goes in the text file is a comma-delimited list of data. Not very user friendly when it comes to troubleshooting time.
• I’ll have to come up with a way to dynamically name the text files with a timestamp, etc., in order to not overwrite the file every day. Otherwise yesterday’s bad data may be overwritten in today’s ETL run.

The Solution

In the engagement described above, I was asked to fix the ETL process in question to make it more robust and not so brittle and prone to failure. The customer also wanted an easier way to identify and analyze the bad data so that end user analysts could take over that task and it would not require developer or BA resources. This is the solution I came up with.

Table

First, we need a table in which to store the bad data. In order to prevent having to have a unique table for each place this is used, I decided to store the data as XML in a single table.  That way, this becomes a generic solution that can be used regardless of the number of columns, data types, etc.

Source code

CREATE TABLE [dbo].[ErrorRows_XML](
	[ErrorID] [INT] IDENTITY(1,1) NOT NULL,
	[ErrorDate] [DATETIME] NOT NULL,
	[ErrorSource] [VARCHAR](500) NULL,
	[ErrorCount] [INT] NULL,
	[RowData]
[xml][/xml]
NULL, [BatchID] [INT] NULL, CONSTRAINT [PK_EFC_ErrorRows_XML] PRIMARY KEY CLUSTERED ([ErrorID] ASC) ) GO

The ErrorSource column is used to identify where in the ETL process this particular error occurred. I used the following naming convention for ErrorSource: PackageName_DataFlowName_ComponentName. So in the example above the ErrorSource would be factSales_Product_GetProductKey.

The ErrorCount column is used to store how many rows are in this particular dataset. Remember that we are storing all the bad data from one given instance of the error in one row via XML.

RowData contains the actual bad data in XML format.

Finally, in this case I have a BatchID column because this ETL process uses batches and it makes it easy to identify which run the bad data came from. This may not be necessary in all uses of this pattern.

The Package

Continuing with our example, let’s modify the package to write the bad data to the table. We’re going to do that by using a Recordset destination. This destination stores the data in a variable for future use.

Here is the new data flow for the product lookup. When this data flow runs, the bad data will be stored as an ADO recordset in a variable.

The recordset destination does not have it’s own GUI so we always use the advanced editor to configure it. But it’s configuration is very simple. The only configuration we have to do is choose the variable to send the output to and choose what columns we want to include in the recordset. This should be a package level variable with a data type of Object.

At this point, we have an SSIS variable that contains the bad data. The last step is to add a Script Task to the package to serialize the data to XML and write it  to the table.

In the script task properties, we include the object variable as a ReadOnly variable.

The script consists of a WriteError() function that does the heavy lifting. It uses the GetXML method of the ADO dataset object to serialize the data as XML. Then it’s a simple ADO call to write the XML data to the database.

Source code

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.Xml;
using System.Data.SqlClient;
using System.Data.OleDb;
using System.IO;
 
namespace ST_d98c4c2b62f94789a154c95495b6588e
{
	[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
	public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
	{
       	public void Main() 
		{
            WriteData("BadData_ProductRecordset_MissingProducts");
			Dts.TaskResult = (int)ScriptResults.Success;
		}
 
 
        public void WriteData(string variable)
        { 
            //serialize to xml
            OleDbDataAdapter da = new OleDbDataAdapter();
            DataTable dt = new DataTable();
            DataSet ds = new DataSet();
 
            da.Fill(dt, Dts.Variables[variable].Value);
            ds.Tables.Add(dt);
            string xml = ds.GetXml();
 
            if (xml != "<NewDataset />")
            {
                //write to db
                string ErrSource = Dts.Variables[variable].Name;
                string BatchID = Dts.Variables["BatchID"].Value.ToString();
                SqlConnection cn = new SqlConnection();
                cn = (SqlConnection)(Dts.Connections["DB1"].AcquireConnection(Dts.Transaction) as SqlConnection);
                SqlCommand cmd = new SqlCommand("insert dbo.ErrorRows_XML(ErrorSource,RowData,BatchID) values (@ErrorSource, @XML, @BatchID)");
                cmd.Connection = cn;
                cmd.Parameters.Add("@ErrorSource", SqlDbType.VarChar).Value = ErrSource;
                cmd.Parameters.Add("@XML", SqlDbType.Xml).Value = xml;
                cmd.Parameters.Add("@BatchID", SqlDbType.Int).Value = BatchID;
                cmd.ExecuteNonQuery();
            }
        }
             }
}

Finally, I create a view for each ErrorSource using XQuery to parse the XML as easy to read data. Ultimately there could be a front end for this in .NET or even SSRS but I haven’t built that.  In any case this should be much more dynamic than having to create a view for each error but I just haven’t invested the time. Even so, having an analyst select from a view instead of poring through text files is much more user friendly.

Source code

CREATE VIEW [dbo].[vBadData_ProductRecordset_MissingProducts]
AS
SELECT 
	x.ErrorID
	,x.ErrorDate
	,x.BatchID
	,tbl.col.VALUE(N'(SalesID)[1]',N'bigint') AS SalesID
	,tbl.col.VALUE(N'(ProductName)[1]',N'varchar(50)') AS ProductName
FROM dbo.ErrorRows_XML x
cross apply RowData.nodes(N'//Table1') AS tbl(col)
WHERE ErrorSource = 'BadData_ProductRecordset_MissingProducts'

The post An SSIS design pattern for capturing and reporting on bad data appeared first on SQL Solutions Group.

The basis of the solution is that we store the data as XML.  This way we maintain the structure and schema and can shred it as needed later on but we don’t have to have schema-specific tables.

To start with, we need a table to store the results.

CREATE TABLE [dbo].[ErrorRows_XML](
    [ErrorID] [int] IDENTITY(1,1) NOT NULL PRIMARY KEY CLUSTERED,
    [ErrorDate] [datetime] NOT NULL,
    [ErrorSource] [varchar](500) NULL,
    [ErrorCount] [int] NULL,
    [RowData] [xml] NULL,
)
GO

The next thing we need to do is create an SSIS variable to store the results.  We are going to use a Recordset destination so we need a variable of type Object to store the Recordset in.

For the variable name, I use the following naming convention:

Error_<PackageName>_<CFStepName>_<DFStepName>

As you can see from the screenshot below, the DataType is Object and the Value is set for you as System.Object.

In this example, I am using the No Match output of a Lookup transform.  If we can’t find the key, we don’t want the row to go to the fact table via the Match output, but we don’t want to lose the data either so we can troubleshoot the key issue.

The destination is a Recordset Destination, which uses the Advanced Editor.  On Component Properties, the only thing we need to specify is the VariableName, which we created in the previous step.

On the Input Columns tab, we specify which columns from the data flow we want to store as XML.  I generally limit this as much as possible to just the data I need to figure out why this particular operation didn’t work.  In this case that would be the primary key of the input for the data flow and the columns used in the lookup.

At this point, when the Data Flow runs, the rows to be logged will be stored in the Recordset variables.  The last step (typically at the end of the control flow) is a Script Task to write the data to the ErrorLog table created above.

The variable we stored the data in needs to be specified in the Script Task (ReadOnly is fine).

The script uses this function to do the work.

 public void WriteError(string variable)
        {
            //serialize to xml
            OleDbDataAdapter da = new OleDbDataAdapter();
            DataTable dt = new DataTable();
            DataSet ds = new DataSet();

            da.Fill(dt, Dts.Variables[variable].Value);
            ds.Tables.Add(dt);
            string xml = ds.GetXml();

            if (xml != "<NewDataSet />")
            {
                //write to db
                string ErrSource = Dts.Variables[variable].Name;
                SqlConnection cn = new SqlConnection();
                cn = (SqlConnection)(Dts.Connections["DW01_ADO"].AcquireConnection(Dts.Transaction) as SqlConnection);
                SqlCommand cmd = new SqlCommand("insert dbo.EFC_ErrorRows_XML(ErrorSource,RowData) values(@ErrorSource, @XML)", cn);
                cmd.Parameters.Add("@ErrorSource", SqlDbType.VarChar).Value = ErrSource;
                cmd.Parameters.Add("@XML", SqlDbType.Xml).Value = xml;
                cmd.ExecuteNonQuery();
            }
        }

The body of the script just calls the function for each variable that contains errors.

 public void Main()
        {
            WriteError("Error_FactEFCTaps_StageTaps_TripKey");
            WriteError("Error_FactEFCTaps_StageTaps_ActionKey");
            WriteError("Error_FactEFCTaps_StageTaps_ServiceKey");
            WriteError("Error_FactEFCTaps_StageTaps_ZoneKey");
            WriteError("Error_FactEFCTaps_StageTaps_YardKey");
            WriteError("Error_FactEFCTaps_StageTaps_MTTapDateKey");
            WriteError("Error_FactEFCTaps_StageTaps_MTTapTimeKey");
            WriteError("Error_FactEFCTaps_StageTaps_StopKeyEndDateNull1");
            WriteError("Error_FactEFCTaps_StageTaps_NoPlatformKey2");
            WriteError("Error_FactEFCTaps_StageTaps_RouteID0");
            WriteError("Error_FactEFCTaps_StageTaps_PlatformKey1");
            WriteError("Error_FactEFCTaps_StageTaps_NoPlatformKey3");
            WriteError("Error_FactEFCTaps_StageTaps_NoPlatformKey1");
            WriteError("Error_FactEFCTaps_StageTaps_AllStopKey");
            WriteError("Error_FactEFCTaps_StageTaps_NoPlatformKey");
            WriteError("Error_FactEFCTaps_StageTaps_InstitutionKey");
            WriteError("Error_FactEFCTaps_StageTaps_ProductKey");
            WriteError("Error_FactEFCTaps_StageTaps_CardKey");
            WriteError("Error_FactEFCTaps_StageTaps_VehicleKeyTraxFR");

            Dts.TaskResult = (int)ScriptResults.Success;
        }

We now have the data in the Error Log table as XML

XQuery can be used to shred the XML to get at specific data.  For common errors, I create views for the specific error type.

SELECT 
    x.ErrorID
    ,x.ErrorDate
    ,tbl.col.value(N'(id_sess)[1]',N'bigint') AS id_sess
    ,tbl.col.value(N'(cardid)[1]',N'nvarchar(255)') AS CardID
    ,tbl.col.value(N'(CC_CompanyName)[1]',N'nvarchar(255)') AS CC_CompanyName
    ,tbl.col.value(N'(CDListMembership)[1]',N'nvarchar(255)') AS CDListMembership
    ,tbl.col.value(N'(FaceNumber)[1]',N'nvarchar(255)') AS FaceNumber
    ,tbl.col.value(N'(id_payee)[1]',N'bigint') AS id_payee
    ,tbl.col.value(N'(LastCDListChangeDate)[1]',N'datetime') AS LastCDListChangeDate
    ,tbl.col.value(N'(ProductKey)[1]',N'int') AS ProductKey
    ,tbl.col.value(N'(ShortName)[1]',N'nvarchar(255)') AS ShortName
    ,tbl.col.value(N'(SubscriptionEndDate)[1]',N'datetime') AS SubscriptionEndDate
    ,tbl.col.value(N'(SubscriptionStartDate)[1]',N'datetime') AS SubscriptionStartDate
    ,tbl.col.value(N'(InstitutionKey)[1]',N'int') AS InstitutionKey
from dbo.EFC_ErrorRows_XML x
cross apply RowData.nodes(N'//Table1') as tbl(col)
WHERE ErrorSource = 'Error_FactEFCTaps_StageTaps_CardKey'

The post Dynamic SSIS Error Logging appeared first on SQL Solutions Group.

    “Provider=OraOLEDB.Oracle;User ID=DBUser;Password=DBPassword;Data Source=OracleSID;”

The package worked flawlessly in development. But when it was deployed to production, it would hang on connections to specific servers. After beating my head against the wall for a while, I discovered that on every server it would fail on, the password was expired for the account. It turns out that by default, the Oracle provider will pop a dialog box for an expired password, prompting a password change. In something batch based like SSIS, this results in the process hanging. Adding a switch to the connection string to suppress the dialog box did the trick.

    “Provider=OraOLEDB.Oracle;User ID=DBUser;Password=DBPassword;Data Source=OracleSID;PwdChgDlg=0;”

This strikes me as something that should default to 0 but in any case it’s one of those little things that is not a big deal until it happens to you and can be tough to track down.

The post Expired Passwords and the Oracle Provider appeared first on SQL Solutions Group.