T-SQL Archives | SQL Solutions Group https://sqlsolutionsgroup.com/tag/t-sql/ Fri, 10 Feb 2023 22:34:24 +0000 en-US hourly 1 https://wordpress.org/?v=6.9.4 https://sqlsolutionsgroup.com/wp-content/uploads/2021/01/cropped-SSG_FAVICON0002-32x32.png T-SQL Archives | SQL Solutions Group https://sqlsolutionsgroup.com/tag/t-sql/ 32 32 T-SQL Tips and Solutions, Part 1 https://sqlsolutionsgroup.com/t-sql-tips/ https://sqlsolutionsgroup.com/t-sql-tips/#comments Thu, 13 May 2021 07:23:00 +0000 https://sqlsolutionsgroup.com/?p=6394 When writing T-SQL code, I find there are certain routines I do over and over again. Like, inserting data into a table, creating temp tables, or searching tables for columns needed in a Join statement. At times like these, it’s good to use T-SQL tips to make my job easier. The small, simple ones are pretty […]

The post T-SQL Tips and Solutions, Part 1 appeared first on SQL Solutions Group.

]]> When writing T-SQL code, I find there are certain routines I do over and over again. Like, inserting data into a table, creating temp tables, or searching tables for columns needed in a Join statement. At times like these, it’s good to use T-SQL tips to make my job easier. The small, simple ones are pretty easy, like inserting data into two or three columns of a table structure that allows all columns to be NULL. In these cases I usually just manually write an insert statement like this one:

 -- Create the Table
 CREATE TABLE A1
 (
   Column1   [INT] IDENTITY(1,1) NOT NULL
  ,Column2   [Varchar](100)      NULL
  ,Column3   [Varchar](100)      NULL
  ,Column4   [Varchar](100)      NULL
  ,Column5   [Varchar](100)      NULL
  ,Column6   [Varchar](100)      NULL
  ,Column7   [Varchar](100)      NULL
  ,Column8   [Varchar](100)      NULL
  ,Column9   [Varchar](100)      NULL
  ,Column10  [Varchar](100)      NULL
  ,Column11  [Varchar](100)      NULL
  ,Column12  [Varchar](100)     NULL
  ,Column13  [Varchar](100)     NULL
  ,Column14  [Varchar](100)     NULL
  ,Column15  [Varchar](100)     NULL
 );
  
  
 --Insert the data
 INSERT A1
 ([Column2], [Column3])
 SELECT 'Column2 data row 1', 'Column3 data row 1'
 UNION
 SELECT 'Column2 data row 2', 'Column3 data row 2'
 UNION
 SELECT 'Column2 data row 3', 'Column3 data row 3';
  
  
 --Select the data as a result set
 SELECT * FROM A1
  
  
 --Drop the table A1
 DROP TABLE A1;

 

And the result set is shown below:

No, the union select to get multiple rows in a single statement is NOT the tip. This is the easy way of inserting into a table where you only have to define two or three columns. But what if we have a different scenario? What if we imported a table exactly like A1 from Server SQLServer01 and we need to copy all the data from the source where column13 = ‘Important’? 

Now we have to create the table again. Set Identity_Insert on for A1, write out the 15 columns in the Insert list, and then copy those columns or write them out again in the select statement. At this point, I like to use a variable that will concatenate the columns for me. I select it out, and then copy and paste the column list instead of typing it out. This comes in handy if you are inserting or creating a number of tables in a stored proc development effort).

First we will create our table again:

USE SSG;
 GO
 -- Create the Table
 CREATE TABLE A1
 (
   Column1   [INT] IDENTITY(1,1) NOT NULL
  ,Column2   Varchar      NULL
  ,Column3   Varchar      NULL
  ,Column4   Varchar      NULL
  ,Column5   Varchar      NULL
  ,Column6   Varchar      NULL
  ,Column7   Varchar      NULL
  ,Column8   Varchar      NULL
  ,Column9   Varchar      NULL
  ,Column10  Varchar      NULL
  ,Column11  Varchar      NULL
  ,Column12  Varchar     NULL
  ,Column13  Varchar     NULL
  ,Column14  Varchar     NULL
  ,Column15  Varchar     NULL
 );

 

We can look at the table definition by looking at the INFORMATION_SCHEMA.COLUMNS table like so:

SELECT *
   FROM INFORMATION_SCHEMA.COLUMNS
  WHERE TABLE_NAME = 'A1';

 

T-SQL Tips

Next we will declare a variable, initialize it as blank in the declare statement, and then use the variable to select from the INFORMATION_SCHEMA.COLUMNS table as shown below:

DECLARE @v_ColumnList  VARCHAR = '';
 SELECT @v_ColumnList = @v_ColumnList + '[' + COLUMN_NAME + '], '
     FROM INFORMATION_SCHEMA.COLUMNS
  WHERE TABLE_NAME = 'A1'
  ORDER BY [ORDINAL_POSITION];
 SELECT @v_ColumnList;

Now I copy the result set and paste it into my insert list and my select criteria after I remove the last comma. In this solution we imported the A1 table into SQLServer02 from SQLServer01 and named the imported table A1_From_SS01. The tip solution looks like:

USE SSG;
 GO
 -- Create the Table
 CREATE TABLE A1
 (
   Column1   [INT] IDENTITY(1,1) NOT NULL
  ,Column2   Varchar      NULL
  ,Column3   Varchar      NULL
  ,Column4   Varchar      NULL
  ,Column5   Varchar      NULL
  ,Column6   Varchar      NULL
  ,Column7   Varchar      NULL
  ,Column8   Varchar      NULL
  ,Column9   Varchar      NULL
  ,Column10  Varchar      NULL
  ,Column11  Varchar      NULL
  ,Column12  Varchar     NULL
  ,Column13  Varchar     NULL
  ,Column14  Varchar     NULL
  ,Column15  Varchar     NULL
 );
 -- Get the column list so you do not have to write it out
 DECLARE @v_ColumnList  VARCHAR = '';
 SELECT @v_ColumnList = @v_ColumnList + '[' + COLUMN_NAME + '], '
     FROM INFORMATION_SCHEMA.COLUMNS
  WHERE TABLE_NAME = 'A1'
  ORDER BY [ORDINAL_POSITION];
 --Select the data as a result set
 SELECT * FROM A1
 SET IDENTITY_INSERT A1 ON
 --Insert the data
 INSERT A1
 (      [Column1], [Column2], [Column3], [Column4], [Column5], [Column6], [Column7], [Column8], [Column9], [Column10], [Column11], [Column12], [Column13], [Column14], [Column15])
 SELECT [Column1], [Column2], [Column3], [Column4], [Column5], [Column6], [Column7], [Column8], [Column9], [Column10], [Column11], [Column12], [Column13], [Column14], [Column15]
   FROM A1_From_SS01
  WHERE [Column13] = 'Important';

 

Summary

In review, the T-SQL tip is the concatenation of the table structure within a select statement that is then just printed or selected from the variable. This in turn is copied and pasted where you need the Column list in your code. This has saved me countless time and illuminated frustration just by eliminating a simple mundane task. We’ll share more T-SQL tips again soon.

The post T-SQL Tips and Solutions, Part 1 appeared first on SQL Solutions Group.

]]> https://sqlsolutionsgroup.com/t-sql-tips/feed/ 1 T-SQL Anti-Patterns: SQL User Defined Functions (UDFs) that turn your set operation into a cursor https://sqlsolutionsgroup.com/t-sql-anti-patterns-user-defined-functions/ https://sqlsolutionsgroup.com/t-sql-anti-patterns-user-defined-functions/#respond Wed, 03 Mar 2021 01:58:09 +0000 https://sqlsolutionsgroup.com/?p=6208 One of the things we see as consultants are consistent patterns of behavior by T-SQL developers that cause big performance problems. I call these anti-patterns. Today I want to specifically address SQL User Defined Functions (UDFs). SQL Server 2000 introduced UDFs. While they can be useful in certain situations, an all too prevalent anti-pattern has […]

The post T-SQL Anti-Patterns: SQL User Defined Functions (UDFs) that turn your set operation into a cursor appeared first on SQL Solutions Group.

]]> One of the things we see as consultants are consistent patterns of behavior by T-SQL developers that cause big performance problems. I call these anti-patterns. Today I want to specifically address SQL User Defined Functions (UDFs).

SQL Server 2000 introduced UDFs. While they can be useful in certain situations, an all too prevalent anti-pattern has emerged.

SQL User Defined Functions: Identifying The Problem

To illustrate this, let’s take a simple example from the AdventureWorks  sample database. If you don’t have this database, you can download and install it. This is a good thing to have on your personal development or lab SQL Server instance, as many blogs, books, and articles use it for examples.

In our example, we are going to generate a simple report for the AdventureWorks Human Resources department using the following tables.

SQL user defined functions

Our report will consist of the employee’s name, the department they work in, their job title, and start date. We have a mock-up of the report that the end user would like to see:

image

Taking a look at the schema above, we realize that the EmployeeDepartmentHistory table has most of this information. So we start there.

SELECT BusinessEntityID AS EmployeeID,
       DepartmentID,
       StartDate
FROM HumanResources.EmployeeDepartmentHistory
image

This gets us the EmployeeID and DepartmentID, but not the actual names. As developers, we are big into encapsulating code as much as possible to avoid repetitive code and the manageability issues that come with it. We know that UDFs in SQL Server can help with this so we create the following function to get the department name:

CREATE FUNCTION dbo.GetDepartment
(
    @DepartmentID INT
)
RETURNS VARCHAR(40)
AS
BEGIN
    DECLARE @ret VARCHAR(40)
    SET @ret =
    (
        SELECT [Name]
        FROM HumanResources.Department
        WHERE DepartmentID = @DepartmentID
    )
    RETURN @ret
END

Now we can execute the same query but call the new function to return the department name.

SELECT
     BusinessEntityID AS EmployeeID,
     dbo. GetDepartment(DepartmentID) AS DepartmentName,
     StartDate
 FROM HumanResources. EmployeeDepartmentHistory
SQL user defined functions

That worked great so we do the same thing for EmployeeName and JobTitle.

CREATE FUNCTION dbo. GetEmployeeName(@EmployeeID int)
RETURNS VARCHAR(40)
AS
BEGIN
  DECLARE @ret VARCHAR(40), @ContactID INT 

  --get contactid
  SELECT @ContactID = BusinessEntityID FROM HumanResources. Employee WHERE BusinessEntityID = @EmployeeID

  --get name
  SET @ret = (	SELECT coalesce(FirstName,'') + ' ' + coalesce(MiddleName,'') + ' ' + coalesce(LastName,'')
				FROM Person. Person
				WHERE BusinessEntityID = @ContactID
			  )

  RETURN @ret
END
CREATE FUNCTION dbo.GetJobTitle
(
    @EmployeeID INT
)
RETURNS VARCHAR(40)
AS
BEGIN
    DECLARE @ret VARCHAR(40)
    SET @ret =
    (
        SELECT [JobTitle]
        FROM HumanResources.Employee
        WHERE BusinessEntityID = @EmployeeID
    )
    RETURN @ret
END

Executing the query again using the three functions we’ve created, we get the data we are looking for. We have encapsulated the code into function for re-use in other places, and we avoided those pesky joins. The resulting query is clean and simple. Best of all, this is a single query so we’re doing a set operation like our DBA has told us we should do.

SELECT
	dbo. GetEmployeeName(BusinessEntityID) as EmployeeName,
	dbo. GetDepartment(DepartmentID) AS DepartmentName,
	dbo. GetJobTitle(BusinessEntityID),
	StartDate
FROM HumanResources. EmployeeDepartmentHistory
SQL user defined functions

So what’s the problem?

The problem is that we have unwittingly turned our single query into a loop by introducing these functions. Because the functions are non-deterministic, they have to be called for each row in our result set.

To see this, let’s fire up our old friend SQL Server Profiler, which shows us the actual executing statements and the accompanying resources used. We’ll use the following trace definition to keep things as simple and clean as possible:

image

Let’s execute our query and see what we get.

SELECT
	dbo. GetEmployeeName(BusinessEntityID) as EmployeeName,
	dbo. GetDepartment(DepartmentID) AS DepartmentName,
	dbo. GetJobTitle(BusinessEntityID),
	StartDate
FROM HumanResources. EmployeeDepartmentHistory
image

Wait a minute. Our single-statement query just generated 2076 individual SQL Statements, generating a total of 2,672 logical reads. If we look through the profiler results carefully, we can see that each of the three functions was called once for each of the 296 rows our query returned. So what looks like a simple set operation is, in fact, a loop.

We go to our DBA for help and she explains that we need to get rid of the functions and use joins instead to make it into a true set operation. So we refactor the query as follows:

SELECT
	COALESCE(c. FirstName,'') + ' ' + coalesce(c. MiddleName,'') + ' ' + coalesce(c. LastName,'') AS EmployeeName,
	d. name AS DepartmentName,
	e. JobTitle,
	StartDate
FROM HumanResources. EmployeeDepartmentHistory eh
	INNER JOIN HumanResources. Department d ON eh. DepartmentID = d. DepartmentID
	INNER JOIN HumanResources. Employee e ON e. BusinessEntityID = eh. BusinessEntityID
	INNER JOIN Person. Person c ON c. BusinessEntityID = e. BusinessEntityID
image

Now if we execute the query and look at the profiler trace, we get just a single statement for a total of 1032 logical reads. Our query has become much more complicated and we don’t have the benefit of code encapsulation, but we’ve got the logical reads to less than half. And that was for only 296 rows.

SQL user defined functions

Conclusion

This is a simple example, but hopefully you can see the impact in a busy system with lots more data. I have seen this in production environments where something like GetCustomerName() is being called tens of millions of times daily.  This is a great example of an anti-pattern, where SQL user defined functions look like a great idea but the performance impact can be devastating.

The post T-SQL Anti-Patterns: SQL User Defined Functions (UDFs) that turn your set operation into a cursor appeared first on SQL Solutions Group.

]]> https://sqlsolutionsgroup.com/t-sql-anti-patterns-user-defined-functions/feed/ 0 Working with the registry in SQL Server https://sqlsolutionsgroup.com/working-registry-sql-server/ https://sqlsolutionsgroup.com/working-registry-sql-server/#comments Thu, 02 Apr 2020 15:16:00 +0000 http://sqlsolutionsgroup.com/?p=2956 There is a lot of information within the Windows registry. Sometimes, it would sure be nice to work with the registry in SQL Server through your T-SQL scripts. As it turns out, there are several undocumented extended stored procedures in SQL Server that do just this. A listing of these procedures are: Regular Instance-Aware sys.xp_regread […]

The post Working with the registry in SQL Server appeared first on SQL Solutions Group.

]]> There is a lot of information within the Windows registry. Sometimes, it would sure be nice to work with the registry in SQL Server through your T-SQL scripts. As it turns out, there are several undocumented extended stored procedures in SQL Server that do just this. A listing of these procedures are:

Regular Instance-Aware
sys.xp_regread sys.xp_instance_regread
sys.xp_regenumvalues sys.xp_instance_regenumvalues
sys.xp_regenumkeys sys.xp_instance_regenumkeys
sys.xp_regwrite sys.xp_instance_regwrite
sys.xp_regdeletevalue sys.xp_instance_regdeletevalue
sys.xp_regdeletekey sys.xp_instance_regdeletekey
sys.xp_regaddmultistring sys.xp_instance_regaddmultistring
sys.xp_regremovemultistring sys.xp_instance_regremovemultistring

As you can see, there are two categories of extended stored procedures you can use, with complimentary procedures between them. The “Instance-Aware” category procedures utilize key word substitution in order to navigate to the appropriate registry keys for the SQL Server instance that is running. I’ll show you an example shortly.

A lot of companies spin their wheels trying to diagnose a SQL problem.

With SQL Server, we’ve seen it all and have done it all. Save time. Call us first.

Learn More

Registry in SQL Server Basics

Before we go too much further, let’s explain a few basic key phrases when dealing with the registry:

Registry: The registry is a hierarchical database used by Windows. It is organized in a tree view manner.

Registry Hive: A hive is a logical group of keys, subkeys, and values in the registry that has a set of supporting files containing backups of its data.

Examples of Registry Hives are:

Registry Hive
HKEY_CURRENT_CONFIG
HKEY_CURRENT_USER
HKEY_LOCAL_MACHINE\SAM
HKEY_LOCAL_MACHINE\Security
HKEY_LOCAL_MACHINE\Software
HKEY_LOCAL_MACHINE\System
HKEY_USERS\.DEFAULT

Of all of these hives, you will probably use HKEY_LOCAL_MACHINE and HKEY_CURRENT_USER the most.

Registry Key: A node within a registry hive is called a key. A key can contain both sub-keys and values. Sometimes an application will only be looking for the presence of a key; at other times it will need to look at the value.

Registry Value: A registry value is simply a piece of data, of any data type.

The following image shows the tree view layout of the registry, and identifies the various items within it:

Registry in SQL Server

Querying the registry from SQL Server

Using xp_regread / xp_instance_regread

Let’s start off by querying some data within SQL Server. I have an instance of SQL Server named “SQL2014” (would you care to take a guess as to what version of SQL Server this is?). One of the items stored in the registry is the location of the SQL Agent working directory. These procedures can query the registry and return the specified values. For example:

EXECUTE master.sys.xp_regread 
    'HKEY_LOCAL_MACHINE', 
    'Software\Microsoft\Microsoft SQL Server\MSSQL12.SQL2014\SQLServerAgent', 
    'WorkingDirectory';

When I execute this statement, SQL returns the following result:

VALUE            DATA
---------------- -----------------------------------
WorkingDirectory D:\MSSQL\MSSQL12.SQL2014\MSSQL\JOBS

In order for you to run this, you may need to change the key as appropriate for the version and instance on your server. The result includes both the value, and the data for the specified path.

xp_instance_regread

In this example, I used xp_regread to read the direct registry path. If you remember from earlier, there are SQL Server instance-aware versions of each registry procedure. A comparable statement using the instance-aware procedure would be:

EXECUTE master.sys.xp_instance_regread
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent',
    'WorkingDirectory';

This statement returns the exact same information. Let’s look at the difference between these – in the first query, the registry path is the exact registry path needed. It includes “\Microsoft SQL Server\MSSQL12.SQL2014\”. In the latter query, this string is replaced with “\MSSQLSERVER\”. Since the latter function is instance aware, it replaces the “MSSQLSERVER” with the exact registry path necessary for this instance of SQL Server. Pretty neat, isn’t it? This allows you to have a script that will run properly regardless of instance. The rest of the examples in this post utilize the instance-aware procedures to make it easier for you to follow along and run these yourself.

Syntax

The syntax for these procedures is:

EXECUTE xp_regread 
    [@rootkey=]’rootkey’,
    [@KEY=]’KEY’
    [, [@value_name=]’value_name’]
    [, [@VALUE=]@VALUE OUTPUT]

The first parameter is the registry hive that you want to query, the second parameter is the key path, and the third is the value name. The third parameter is optional – if provided, then the procedure will return the data from the specified value item; if not provided, then the procedure only returns whether the specified key exists. There is also an optional fourth parameter, which is an output parameter, and the data output goes into that. The parameters are positional, and while you can specify a name for the parameter, any name will work. Since the parameters are positional, my recommendation is to not use the parameter names. The following example utilizes the optional fourth parameter, and it will return just the specified path into the variable:

DECLARE @SQLAgentDirectory VARCHAR(255);
EXECUTE master.sys.xp_instance_regread
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent',
    'WorkingDirectory',
    @SQLAgentDirectory OUTPUT;
SELECT @SQLAgentDirectory;

You would think that if you specify either NULL or an empty string for the third parameter, that you could send whether the key exists to an output variable. However, I have not found a way to accomplish this. Specifying either of these values results in an error when running this statement. If you know how to do this, please leave a remark so that this post can be updated with that information.

Using xp_regenumvalues / xp_instance_regenumvalues

These procedures enumerate through all of the values of the specified key, returning a separate result set for each value. For instance, the following statement will return all of the values in the above SQLServerAgent key:

EXECUTE master.sys.xp_instance_regenumvalues
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent';

Having all of these result sets makes it difficult to work with this procedure. Thankfully, you can put these all into one result set. Just create a temporary table (or table variable) to hold the output, and then use INSERT / EXECUTE to fill it, like the following example does:

DECLARE @Registry TABLE (VALUE VARCHAR(255), DATA VARCHAR(255));
INSERT INTO @Registry
EXECUTE master.sys.xp_instance_regenumvalues
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent';
SELECT * FROM @Registry;

Now that all of the results are in one result set, you can work with it a bit easier.

Syntax

The syntax for these procedures is:

EXECUTE xp_regenumvalues 
    [@rootkey=]’rootkey’,
    [@key=]’key’

Using xp_regenumkeys / xp_instance_regenumkeys

These procedures enumerate through all of the keys in a specified path, and return all of the keys in that path. Unlike xp_instance_regenumvalues, all of the keys are returned in one result set, though you will probably want to use INSERT / EXECUTE to put this into temporary storage so that you can work with it. An example of using these procedures is:

EXECUTE master.sys.xp_instance_regenumkeys
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER';

The syntax for these procedures is:

EXECUTE xp_regenumkeys 
    [@rootkey=]’rootkey’,
    [@KEY=]’KEY’

Modifying the registry from within SQL Server

Up to this point, we have focused on retrieving data from the registry. What if you want to modify the registry? Read on… be forewarned that the following procedures are modifying the registry, which means that they can also damage the registry, possibly rendering the server unusable. Use at your own risk!

xp_regwrite / xp_instance_regwrite

Use these procedures to create keys and write data into the registry. You can create up to 32 sub-keys at a time. In the following example, a new key “MyNewKey” will be added to the SQLServerAgent key, and the value “MyNewValue” will be added to this new key with the data “Now you see me!”. It will then read this value from the registry.

Syntax

The syntax for these procedures is:

EXECUTE xp_regwrite 
    [@rootkey=]’rootkey’,
    [@KEY=]’KEY’,
    [@value_name=]’value_name’,
    [@type=]’type’,
    [@VALUE=]’VALUE’

xp_regdeletevalue / xp_instance_regdeletevalue

These procedures are used to delete a specified value from the registry. In this example, the “MyNewValue” value will be deleted. The example first enumerates through all of the values in this key (just the one), deletes the value, enumerates through them again (since there are no values, there will be no result set), and then finally shows that the key is still present.

EXECUTE master.sys.xp_instance_regenumvalues
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey';
EXECUTE master.sys.xp_instance_regdeletevalue
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey',
    'MyNewValue';
EXECUTE master.sys.xp_instance_regenumvalues
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey';
EXECUTE master.sys.xp_instance_regread
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey';

Syntax

The syntax for these procedures is:

EXECUTE xp_regdeletevalue 
    [@rootkey=]’rootkey’,
    [@KEY=]’KEY’,
    [@value_name=]’value_name’

xp_regdeletekey / xp_instance_regdeletekey

Use these procedures to delete an entire key from the registry. In this example, the script first adds another new key under “MyNewKey”, and a value in that new key. The script then deletes both the “AnotherNewKey” (which deletes the value just added also) and “MyNewKey” keys and finally shows that both keys have been deleted. Note that in order to delete a key, it cannot have any sub-keys, which is why the script deletes “AnotherNewKey” first (try running the script first by commenting out the first xp_instance_regdeletekey to see that it must be empty).

EXECUTE master.sys.xp_instance_regwrite
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey\AnotherNewKey',
    'MyNewValue',
    'REG_SZ',
    'Another new value!';
EXECUTE master.sys.xp_instance_regread
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey\AnotherNewKey';
EXECUTE master.sys.xp_instance_regdeletekey
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey\AnotherNewKey';
EXECUTE master.sys.xp_instance_regdeletekey
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey';
EXECUTE master.sys.xp_instance_regread
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey\AnotherNewKey';
EXECUTE master.sys.xp_instance_regread
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey';

Syntax

The syntax for these procedures is:

EXECUTE xp_regdeletekey 
    [@rootkey=]’rootkey’,
    [@KEY=]’KEY’

xp_regaddmultistring / xp_instance_regaddmultistring

Use these procedures to add a string to a multi-string entry in the registry, or to create a multi-string registry entry. In this example, I’ll call the procedure twice. The first time will create the entry with one string in it, and the second time will add a second string to it. Then the example will show the results of this value.

EXECUTE master.sys.xp_instance_regaddmultistring
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey',
    'MyNewValue',
    'A multi-string value!';
EXECUTE master.sys.xp_instance_regaddmultistring
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey',
    'MyNewValue',
    'Yet Another new string added to this multi-string value!';
EXECUTE master.sys.xp_instance_regread
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey',
    'MyNewValue'

Did you notice that the regread procedure got a little confused here? It has two value columns (which is the name of the value), but the second one has the data for the first one. Then the data column is null. If a third string is added, it still returns just these three columns. To see that we actually added these strings, we’ll have to use regedit.exe:

-- add a third string
EXECUTE master.sys.xp_instance_regaddmultistring
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey',
    'MyNewValue',
    'How about a third string?';
-- only shows the first string
EXECUTE master.sys.xp_instance_regread
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey',
    'MyNewValue'

Registry2

Syntax

The syntax for these procedures is:

EXECUTE xp_regaddmultistring 
    [@rootkey=]’rootkey’,
    [@KEY=]’KEY’,
    [@value_name=]’value_name’,
    [@VALUE=]’VALUE’

xp_regremovemultistring / xp_instance_regremovemultistring

These procedures are used to remove a string from a multi-string entry. This example will remove the middle string.

EXECUTE master.sys.xp_instance_regremovemultistring
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey',
    'MyNewValue',
    'Yet Another new string added to this multi-string value!';

Examining the string now in regedit.exe, we can see it removed all of the strings starting with the specified string… in other words, it removed the second and third strings. If we add the strings back in, and then remove the third string, we can see that it also removes the second and third string. In testing with a fourth string, it appears that if you are deleting a string that is not the first string, then all of the remainder of the strings after the first string are removed. Deleting the first string deletes all of the strings. Well, these are undocumented procedures, so it’s not likely that this bug will ever be fixed.

Syntax

The syntax for these procedures is:

EXECUTE xp_regremovemultistring 
    [@rootkey=]’rootkey’,
    [@KEY=]’KEY’,
    [@value_name=]’value_name’,
    [@VALUE=]’VALUE’

And finally, let’s ensure that things are cleaned up:

EXECUTE master.sys.xp_instance_regdeletekey
    'HKEY_LOCAL_MACHINE',
    'Software\Microsoft\MSSQLSERVER\SQLServerAgent\MyNewKey';

Conclusion

With the exception of removing and reading the multi-string values, these extended stored procedures all work pretty well. We’ve been able to read registry values, and to enumerate through a list of keys and values. We can create and delete keys and values.

If you try to use the procedures in sections outside of SQL Server keys, you may run into registry security problems where the SQL Server service account doesn’t have permissions to accomplish the task. On the internet, you will find articles telling you to add the service account to the local administrators group (and to restart the service). This is a VERY BAD IDEA – by doing this, you can completely compromise the server. You could do anything that you desire on the server. Instead, use regedt32.exe, which will allow you to modify the permissions on the key that where you are having access issues.

I close by repeating my warning from above – be very careful when modifying the registry. It is possible to corrupt the registry to the point where the server will no longer function. To recover, an operating system reinstall will be necessary.

As a footnote on working with the instance-aware installation. I recall seeing on the internet other keywords that could be substituted instead of MSSQLSERVER. One of them is SQLServerAgent. However, I can’t find any others right now. If you happen to know of other keywords that work, please reply to this post so that it can be added to this post.

The post Working with the registry in SQL Server appeared first on SQL Solutions Group.

]]> https://sqlsolutionsgroup.com/working-registry-sql-server/feed/ 2 The Finished Product: Framework, pt. 5 https://sqlsolutionsgroup.com/t-sql-framework-2/ https://sqlsolutionsgroup.com/t-sql-framework-2/#respond Tue, 03 Sep 2019 15:15:11 +0000 http://sqlsolutionsgroup.com/?p=4257 We now have a finished T-SQL framework, and this is the last post of the series. We start with the complete Template (with the error engine, activity log and batching). In this post, we’ll examine each new segment and focus on the batching feature, explain what is going on and the necessity. The finished Template: […]

The post The Finished Product: Framework, pt. 5 appeared first on SQL Solutions Group.

]]> We now have a finished T-SQL framework, and this is the last post of the series. We start with the complete Template (with the error engine, activity log and batching). In this post, we’ll examine each new segment and focus on the batching feature, explain what is going on and the necessity.

The finished Template:

USE DBCATCH
GO
IF EXISTS(SELECT 1
            FROM [dbo].[sysobjects] (NOLOCK)
           WHERE [id]                                            = OBJECT_ID(N'[dbo].[<Name, SYSNAME, PU_DB_ADD>]')
             AND OBJECTPROPERTY([id], N'IsProcedure')            = 1) 
BEGIN 
  DROP PROCEDURE [dbo].[<Name, SYSNAME, PU_DB_ADD>];
 
  PRINT '>>> DROPPED PROCEDURE [dbo].[<Name, SYSNAME, PU_DB_ADD>] <<<';
END 
GO
CREATE PROC [dbo].[<Name, SYSNAME, PU_DB_ADD>]
  @p_Batch_ID                        [UNIQUEIDENTIFIER]          = NULL
WITH ENCRYPTION
AS
/*************************************************************************************************/ 
/* Name        : <Name, SYSNAME, PU_DB_ADD>                                                      */
/* Version     : 1.0                                                                             */
/* Author      : Jared Kirkpatrick                                                               */
/* Date        : 2019-05-24                                                                      */
/* Description :                                                                                 */
/*************************************************************************************************/ 
/* Date        : Version: Who: Description                                                       */
/*************************************************************************************************/ 
/* 2019-05-24  : 1.0    : REB: Initial Release.                                                */
/*************************************************************************************************/ 
SET NOCOUNT ON;
/* Declare Variables */
DECLARE @v_Error                     [INT]
       ,@v_MSG                       [NVARCHAR](500)
       ,@v_DB                        [NVARCHAR](100)
       ,@v_Obj                       [NVARCHAR](100)
       ,@v_App                       [NVARCHAR](100)
       ,@v_User                      [NVARCHAR](100)
       ,@v_Spid                      [NVARCHAR](100)
       ,@v_ActivityLog_ID            [INT]
       ,@v_ADesc                     [NVARCHAR](512)
       ,@v_BatchStatus               [BIT]
       ,@v_Now                       [DATETIME]
       ,@v_LastErrorDate             [DATETIME]
       ,@v_SystemName                [NVARCHAR](100)
       ,@v_RowCount                  [INT]
       ,@v_Count                     [INT]
       ,@v_MinINT                    [INT]
       ,@v_MaxINT                    [INT]
       ,@v_Version                   [INT]       
       ,@v_KeyWord                   [SYSNAME]   
       ,@v_MSSQL_ID                  [INT]
       ,@v_Domain                    [NVARCHAR](100)  
       ,@v_Batch_Type_ID             [INT] 
	   ,@v_System                    [NVARCHAR](100)
       ,@v_SQL                       [NVARCHAR](4000)  
       ,@v_UserName                  [NVARCHAR](100)
       ,@v_PW                        [NVARCHAR](100)
       ,@v_Debug                     [INT];
 
SELECT @v_Now                        = GETDATE()
      ,@v_DB                         = DB_NAME()
      ,@v_Obj                        = '[DBCATCH].[dbo].[<Name, SYSNAME, PU_DB_ADD>]'
      ,@v_App                        = APP_NAME()
      ,@v_User                       = ISNULL(ORIGINAL_LOGIN(), USER_NAME()) 
      ,@v_Spid                       = CONVERT([NVARCHAR](25), @@SPID)
	  ,@v_Batch_Type_ID              = [dbo].[FSU_Code_GET]('Batch', 'PROCEDURE')
	  ,@v_System                     = HOST_NAME()
      ,@v_Debug                      = 0;  
 
 
/* Create Temp Tables */
 
IF @v_Debug                          = 1
BEGIN -- debug = 1 Ref: 1
  SELECT 'REF:1 - Define Static Variables';
  SELECT @v_Now                      AS [GETDATE]
        ,@v_DB                       AS [DB]
        ,@v_Obj                      AS [OBJ]
        ,@v_App                      AS [APP]
        ,@v_User                     AS [USER]
        ,@v_Spid                     AS [SPID]
        ,@p_Batch_ID                 AS [Batch_ID];
 
END; -- debug = 1    
 
/* Start the activity_log */
EXEC [DBCATCH].[dbo].[PU_ActivityLog_ADD]
  @p_Error_ID                        = NULL,
  @p_System                          = @v_System,
  @p_DB                              = @v_DB,
  @p_Obj                             = @v_Obj,
  @p_App                             = @v_App,
  @p_User                            = @v_User,
  @p_Spid                            = @v_Spid,
  @p_Description                     = NULL,
  @p_StartTime                       = @v_Now,
  @p_EndTime                         = NULL,
  @p_ActivityLog_ID                  = @v_ActivityLog_ID OUTPUT;     
 
-- Start the batch
IF @p_Batch_ID IS NULL
BEGIN  -- Create the batch
  SELECT @v_BatchStatus              = 1;
  EXEC [DBCATCH].[dbo].[PU_Batch_ADD]
    @p_Batch_ID                      = @p_Batch_ID OUTPUT
   ,@p_Type_ID                       = @v_Batch_Type_ID
   ,@p_Name                          = @v_Obj
   ,@p_Parent_ID                     = @v_ActivityLog_ID;
 
END;   -- Create the batch   
-- Add the next record to the batch
EXEC [DBCATCH].[dbo].[PU_BatchDetail_ADD]
  @p_Batch_ID                        = @p_Batch_ID,
  @p_Object_ID                       = @v_ActivityLog_ID,
  @p_Object                          = @v_Obj;           
  BEGIN TRY
 
--==>> Start Code Here!
  END TRY
 
  BEGIN CATCH
 
    SELECT @v_Error                  = 100000
          ,@v_MSG                    = CONVERT(NVARCHAR(2500), 
                                       'Error: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_NUMBER(), -1)) + 
                                       ' Severity: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_SEVERITY(), -1)) +
                                       ' State: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_STATE(), -1)) +
                                       ' Line: ' +  CONVERT([NVARCHAR](255), ISNULL(ERROR_LINE(), -1)) +
                                       ' Procedure: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_PROCEDURE(), @v_Obj)) +
                                       ' MSG: Error from [<Name, SYSNAME, PU_DB_ADD>]  with error:' + ISNULL(ERROR_MESSAGE(), ''));  
    EXEC [DBCATCH].[dbo].[PU_ErrorLog_ADD]
      @p_Error_ID                    = @v_Error,
      @p_System                      = @v_System,
      @p_DB                          = @v_DB,
      @p_Obj                         = @v_Obj,
      @p_App                         = @v_App,
      @p_User                        = @v_User,
      @p_ErrorMsg                    = @v_MSG;
 
    EXEC [DBCATCH].[dbo].[PU_ActivityLog_UPD]
      @p_ActivityLog_ID              = @v_ActivityLog_ID,
      @p_EndTime                     = @v_Now,
      @p_Error_ID                    = @v_Error,
      @p_Active                      = 0,
      @p_Description                 = @v_MSG,
      @p_DateModified                = @v_Now;    
    EXEC [DBCATCH].[dbo].[PU_BatchDetail_UPD]
      @p_ActivityLog_ID              = @v_ActivityLog_ID
     ,@p_Batch_ID                    = @p_Batch_ID
     ,@p_Now                         = @v_Now;         
    RETURN @v_Error;
 
  END CATCH;
EOF: 
/* Cleanup */
/* Update the activity Log */
EXEC [DBCATCH].[dbo].[PU_ActivityLog_UPD]
  @p_ActivityLog_ID                  = @v_ActivityLog_ID,
  @p_EndTime                         = @v_Now,
  @p_Active                          = 0,
  @p_DateModified                    = @v_Now;
 
/* Update the batch detail */
EXEC [DBCATCH].[dbo].[PU_BatchDetail_UPD]
  @p_ActivityLog_ID                  = @v_ActivityLog_ID
 ,@p_Batch_ID                        = @p_Batch_ID
 ,@p_Now                             = @v_Now;  
 
IF @v_BatchStatus                    = 1
BEGIN
  UPDATE [DBCATCH].[dbo].[TU_Batch]     WITH (ROWLOCK)
     SET [Active]                    = 0
	    ,[DateModified]              = GETDATE()
   WHERE [Batch_ID]                  = @p_Batch_ID;
END;  
GO   
IF OBJECT_ID('[dbo].[<Name, SYSNAME, PU_DB_ADD>]') IS NOT NULL
BEGIN
	PRINT '>>> CREATED PROCEDURE [dbo].[<Name, SYSNAME, PU_DB_ADD>] <<<';
END;
ELSE 
BEGIN
	PRINT '>>> FAILED TO CREATE PROCEDURE [dbo].[<Name, SYSNAME, PU_DB_ADD>] <<<';
END;
GO

The Input Parameter

You’ll notice first that our Template has an input parameter called @p_Batch_ID and it is set to NULL.

a finished t-sql framework

You will see soon that if the @p_Batch_ID is NULL. This means it will be the first stored procedure in the batch. It may be the only stored proc that will execute, so that batch will only have a single execution. Otherwise, the parameter will be set and passed down to the nested procedures appropriately. You will see this in action when we get to our demos.

Next, we go directly under our activity log Insert and add the logic to check the @p_Batch_ID param:

If the @p_Batch_ID was passed in and not null, this would not be the first procedure in the batch. It would skip this part of the logic if branch. But when it is the first procedure that will call other procs (we call this a wrapper procedure), we first set a variable called @v_BatchStatus to 1. This only remains in the scope of the current procedure and will not apply down the nested execution path.

When all the procedures nested have executed, the wrapper will come back to scope and we will check this variable. We’ll cover this more later when we get there. Now we will have a @p_Batch_ID and a value for @v_BatchStatus. We’ll add a record to the Batch Detail even if this is the only procedure that will execute.

Looking at the PU_BatchDetail_ADD proc, we see that we are passing in the @p_Batch_ID parameter that we defined in the PU_Batch_ADD call. The @v_ActivityLog_ID is stuffed into the object_id (so we need more detail about the execution of this procedure we can go directly to the activity log), and the @v_Obj variable that we defined with our other scope variables is stuffed into the Object column. This allows us to group procedures in a wrapper/process while leveraging our other features. Benefit: quick access to more detailed information in the activity logs.

This completes the initial insertion of our batching process for grouping stored procedures. Now we will continue down the template where we update the detail records as we get close to the end of the procedure execution.

our glorious t-sql framework

Towards the end of the TRY CATCH logic, if there was an error, a call is made to the PU_BatchDetail_UPD procedure with the activitylog_ID and the Batch_ID Param. This enables the update to use those values in a distinct where clause, where it will close the detail record with an end time, and the error will be recorded in the error log and activity log accordingly.

When the procedure executes successfully without an error, the same call is executed to the PU_BatchDetail_UPD procedure with the same parameters. We can expect corelating records in the error log and activity log will be absent an error as we would hope all executions would exit successfully.

Finally, we come to some logic that we touched on before where we evaluate the @v_BatchStatus variable:

When we have executed all the nested procedures, control will bubble back up to the original procedure, the wrapper. The wrapper sets the batchstatus, so when the if logic checks equates to 1, we know we are at the wrapper and we can close off the Batch. We do this by setting the active flag to 0 and updating the datemodified column to GETDATE().

The Demo Stage

That summarizes the Template of our T-SQL framework, which means we are code complete and ready to demo. Well … not quite ready to demo. We need to clean up our Error Log and Activity Log tables so they are new and ready for clean inserts.

The first table we clear is the TU_ActivityLog table by executing:

TRUNCATE TABLE TU_ActivityLog;

Next we do the same thing with the TU_Errorlog table:

TRUNCATE TABLE TU_ErrorLog;

Last we check the two tables to make sure they are clear:

Now we can create some demo procedures to test and validate our T-SQL framework. For this demo, we will create a wrapper procedure called PU_Demo1. Yes, this was the same procedure in the last two features. We will drop and then recreate the PU_Demo1 Routine so it will be more clean without having too many demo procedures lagging behind. Plus, add two nested child procedures called PU_Demo2 and PU_Demo3.

  1. PU_Demo1 – the Wrapper will print PU_Demo1 and then call procedures PU_Demo2 and PU_Demo3
  2. PU_Demo2 – the first nested procedure will print PU_Demo2 and exit back to the wrapper
  3. PU_Demo3 – the final nested procedure will print PU_Demo3, then raise an error, and then exit back to the initiating wrapper procedure.

In this case it will be easier to work backwards and develop PU_Demo3, then PU_Demo2, and last we will create PU_Demo1. Start with the code below, defining PU_Demo3. We copy our template code to a new query window and then press CTRL+SHIFT+M and type PU_Demo3 in the object line. We replace the start code here with:

SELECT 'This is PU_Demo3'
    RAISERROR ('Message', 19, 1) WITH LOG

The PU_Demo3 procedure is available in its whole below for completeness.

USE DBCATCH
GO
IF EXISTS(SELECT 1
            FROM [dbo].[sysobjects] (NOLOCK)
           WHERE [id]                                            = OBJECT_ID(N'[dbo].[PU_Demo3]')
             AND OBJECTPROPERTY([id], N'IsProcedure')            = 1) 
BEGIN 
  DROP PROCEDURE [dbo].[PU_Demo3];
 
  PRINT '>>> DROPPED PROCEDURE [dbo].[PU_Demo3] <<<';
END 
GO
CREATE PROC [dbo].[PU_Demo3]
  @p_Batch_ID                        [UNIQUEIDENTIFIER]          = NULL
WITH ENCRYPTION
AS
/*************************************************************************************************/ 
/* Name        : PU_Demo3                                                      */
/* Version     : 1.0                                                                             */
/* Author      : Jared Kirkpatrick                                                               */
/* Date        : 2019-08-24                                                                      */
/* Description :                                                                                 */
/*************************************************************************************************/ 
/* Date        : Version: Who: Description                                                       */
/*************************************************************************************************/ 
/* 2019-08-24  : 1.0    : JSK: Initial Release.                                                */
/*************************************************************************************************/ 
SET NOCOUNT ON;
/* Declare Variables */
DECLARE @v_Error                     [INT]
       ,@v_MSG                       [NVARCHAR](500)
       ,@v_DB                        [NVARCHAR](100)
       ,@v_Obj                       [NVARCHAR](100)
       ,@v_App                       [NVARCHAR](100)
       ,@v_User                      [NVARCHAR](100)
       ,@v_Spid                      [NVARCHAR](100)
       ,@v_ActivityLog_ID            [INT]
       ,@v_ADesc                     [NVARCHAR](512)
       ,@v_BatchStatus               [BIT]
       ,@v_Now                       [DATETIME]
       ,@v_LastErrorDate             [DATETIME]
       ,@v_SystemName                [NVARCHAR](100)
       ,@v_RowCount                  [INT]
       ,@v_Count                     [INT]
       ,@v_MinINT                    [INT]
       ,@v_MaxINT                    [INT]
       ,@v_Version                   [INT]       
       ,@v_KeyWord                   [SYSNAME]   
       ,@v_MSSQL_ID                  [INT]
       ,@v_Domain                    [NVARCHAR](100)  
       ,@v_Batch_Type_ID             [INT] 
	   ,@v_System                    [NVARCHAR](100)
       ,@v_SQL                       [NVARCHAR](4000)  
       ,@v_UserName                  [NVARCHAR](100)
       ,@v_PW                        [NVARCHAR](100)
       ,@v_Debug                     [INT];
 
SELECT @v_Now                        = GETDATE()
      ,@v_DB                         = DB_NAME()
      ,@v_Obj                        = '[DBCATCH].[dbo].[PU_Demo3]'
      ,@v_App                        = APP_NAME()
      ,@v_User                       = ISNULL(ORIGINAL_LOGIN(), USER_NAME()) 
      ,@v_Spid                       = CONVERT([NVARCHAR](25), @@SPID)
	  ,@v_Batch_Type_ID              = [dbo].[FSU_Code_GET]('Batch', 'PROCEDURE')
	  ,@v_System                     = HOST_NAME()
      ,@v_Debug                      = 0;  
 
 
/* Create Temp Tables */
 
IF @v_Debug                          = 1
BEGIN -- debug = 1 Ref: 1
  SELECT 'REF:1 - Define Static Variables';
  SELECT @v_Now                      AS [GETDATE]
        ,@v_DB                       AS [DB]
        ,@v_Obj                      AS [OBJ]
        ,@v_App                      AS [APP]
        ,@v_User                     AS [USER]
        ,@v_Spid                     AS [SPID]
        ,@p_Batch_ID                 AS [Batch_ID];
 
END; -- debug = 1    
 
/* Start the activity_log */
EXEC [DBCATCH].[dbo].[PU_ActivityLog_ADD]
  @p_Error_ID                        = NULL,
  @p_System                          = @v_System,
  @p_DB                              = @v_DB,
  @p_Obj                             = @v_Obj,
  @p_App                             = @v_App,
  @p_User                            = @v_User,
  @p_Spid                            = @v_Spid,
  @p_Description                     = NULL,
  @p_StartTime                       = @v_Now,
  @p_EndTime                         = NULL,
  @p_ActivityLog_ID                  = @v_ActivityLog_ID OUTPUT;     
 
-- Start the batch
IF @p_Batch_ID IS NULL
BEGIN  -- Create the batch
  SELECT @v_BatchStatus              = 1;
  EXEC [DBCATCH].[dbo].[PU_Batch_ADD]
    @p_Batch_ID                      = @p_Batch_ID OUTPUT
   ,@p_Type_ID                       = @v_Batch_Type_ID
   ,@p_Name                          = @v_Obj
   ,@p_Parent_ID                     = @v_ActivityLog_ID;
 
END;   -- Create the batch   
-- Add the next record to the batch
EXEC [DBCATCH].[dbo].[PU_BatchDetail_ADD]
  @p_Batch_ID                        = @p_Batch_ID,
  @p_Object_ID                       = @v_ActivityLog_ID,
  @p_Object                          = @v_Obj;           
  BEGIN TRY
 
SELECT 'This is PU_Demo3'
    RAISERROR ('Message', 19, 1) WITH LOG
  END TRY
 
  BEGIN CATCH
 
    SELECT @v_Error                  = 100000
          ,@v_MSG                    = CONVERT(NVARCHAR(2500), 
                                       'Error: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_NUMBER(), -1)) + 
                                       ' Severity: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_SEVERITY(), -1)) +
                                       ' State: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_STATE(), -1)) +
                                       ' Line: ' +  CONVERT([NVARCHAR](255), ISNULL(ERROR_LINE(), -1)) +
                                       ' Procedure: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_PROCEDURE(), @v_Obj)) +
                                       ' MSG: Error from PU_Demo3 with error:' + ISNULL(ERROR_MESSAGE(), ''));  
    EXEC [DBCATCH].[dbo].[PU_ErrorLog_ADD]
      @p_Error_ID                    = @v_Error,
      @p_System                      = @v_System,
      @p_DB                          = @v_DB,
      @p_Obj                         = @v_Obj,
      @p_App                         = @v_App,
      @p_User                        = @v_User,
      @p_ErrorMsg                    = @v_MSG;
 
    EXEC [DBCATCH].[dbo].[PU_ActivityLog_UPD]
      @p_ActivityLog_ID              = @v_ActivityLog_ID,
      @p_EndTime                     = @v_Now,
      @p_Error_ID                    = @v_Error,
      @p_Active                      = 0,
      @p_Description                 = @v_MSG,
      @p_DateModified                = @v_Now;    
    EXEC [DBCATCH].[dbo].[PU_BatchDetail_UPD]
      @p_ActivityLog_ID              = @v_ActivityLog_ID
     ,@p_Batch_ID                    = @p_Batch_ID
     ,@p_Now                         = @v_Now;         
    RETURN @v_Error;
 
  END CATCH;
EOF: 
/* Cleanup */
/* Update the activity Log */
EXEC [DBCATCH].[dbo].[PU_ActivityLog_UPD]
  @p_ActivityLog_ID                  = @v_ActivityLog_ID,
  @p_EndTime                         = @v_Now,
  @p_Active                          = 0,
  @p_DateModified                    = @v_Now;
 
/* Update the batch detail */
EXEC [DBCATCH].[dbo].[PU_BatchDetail_UPD]
  @p_ActivityLog_ID                  = @v_ActivityLog_ID
 ,@p_Batch_ID                        = @p_Batch_ID
 ,@p_Now                             = @v_Now;  
 
IF @v_BatchStatus                    = 1
BEGIN
  UPDATE [DBCATCH].[dbo].[TU_Batch]     WITH (ROWLOCK)
     SET [Active]                    = 0
	    ,[DateModified]              = GETDATE()
   WHERE [Batch_ID]                  = @p_Batch_ID;
END;  
GO   
IF OBJECT_ID('[dbo].[PU_Demo3]') IS NOT NULL
BEGIN
	PRINT '>>> CREATED PROCEDURE [dbo].[PU_Demo3] <<<';
END;
ELSE 
BEGIN
	PRINT '>>> FAILED TO CREATE PROCEDURE [dbo].[PU_Demo3] <<<';
END;
GO

The next procedure PU_Demo2 we will copy the template code to a new query window, press CTRL+SHIFT+M, and type the name PU_Demo2. Go down to the middle of the procedure where it says:

==>> Start Code Here!

And replace that with:

SELECT 'This is PU_Demo2'

The complete code for PU_Demo2 is below:

USE DBCATCH
GO
IF EXISTS(SELECT 1
            FROM [dbo].[sysobjects] (NOLOCK)
           WHERE [id]                                            = OBJECT_ID(N'[dbo].[PU_Demo2]')
             AND OBJECTPROPERTY([id], N'IsProcedure')            = 1) 
BEGIN 
  DROP PROCEDURE [dbo].[PU_Demo2];
 
  PRINT '>>> DROPPED PROCEDURE [dbo].[PU_Demo2] <<<';
END 
GO
CREATE PROC [dbo].[PU_Demo2]
  @p_Batch_ID                        [UNIQUEIDENTIFIER]          = NULL
WITH ENCRYPTION
AS
/*************************************************************************************************/ 
/* Name        : PU_Demo2                                                      */
/* Version     : 1.0                                                                             */
/* Author      : Jared Kirkpatrick                                                               */
/* Date        : 2019-05-24                                                                      */
/* Description :                                                                                 */
/*************************************************************************************************/ 
/* Date        : Version: Who: Description                                                       */
/*************************************************************************************************/ 
/* 2019-05-24  : 1.0    : JSK: Initial Release.                                                */
/*************************************************************************************************/ 
SET NOCOUNT ON;
/* Declare Variables */
DECLARE @v_Error                     [INT]
       ,@v_MSG                       [NVARCHAR](500)
       ,@v_DB                        [NVARCHAR](100)
       ,@v_Obj                       [NVARCHAR](100)
       ,@v_App                       [NVARCHAR](100)
       ,@v_User                      [NVARCHAR](100)
       ,@v_Spid                      [NVARCHAR](100)
       ,@v_ActivityLog_ID            [INT]
       ,@v_ADesc                     [NVARCHAR](512)
       ,@v_BatchStatus               [BIT]
       ,@v_Now                       [DATETIME]
       ,@v_LastErrorDate             [DATETIME]
       ,@v_SystemName                [NVARCHAR](100)
       ,@v_RowCount                  [INT]
       ,@v_Count                     [INT]
       ,@v_MinINT                    [INT]
       ,@v_MaxINT                    [INT]
       ,@v_Version                   [INT]       
       ,@v_KeyWord                   [SYSNAME]   
       ,@v_MSSQL_ID                  [INT]
       ,@v_Domain                    [NVARCHAR](100)  
       ,@v_Batch_Type_ID             [INT] 
	   ,@v_System                    [NVARCHAR](100)
       ,@v_SQL                       [NVARCHAR](4000)  
       ,@v_UserName                  [NVARCHAR](100)
       ,@v_PW                        [NVARCHAR](100)
       ,@v_Debug                     [INT];
 
SELECT @v_Now                        = GETDATE()
      ,@v_DB                         = DB_NAME()
      ,@v_Obj                        = '[DBCATCH].[dbo].[PU_Demo2]'
      ,@v_App                        = APP_NAME()
      ,@v_User                       = ISNULL(ORIGINAL_LOGIN(), USER_NAME()) 
      ,@v_Spid                       = CONVERT([NVARCHAR](25), @@SPID)
	  ,@v_Batch_Type_ID              = [dbo].[FSU_Code_GET]('Batch', 'PROCEDURE')
	  ,@v_System                     = HOST_NAME()
      ,@v_Debug                      = 0;  
 
 
/* Create Temp Tables */
 
IF @v_Debug                          = 1
BEGIN -- debug = 1 Ref: 1
  SELECT 'REF:1 - Define Static Variables';
  SELECT @v_Now                      AS [GETDATE]
        ,@v_DB                       AS [DB]
        ,@v_Obj                      AS [OBJ]
        ,@v_App                      AS [APP]
        ,@v_User                     AS [USER]
        ,@v_Spid                     AS [SPID]
        ,@p_Batch_ID                 AS [Batch_ID];
 
END; -- debug = 1    
 
/* Start the activity_log */
EXEC [DBCATCH].[dbo].[PU_ActivityLog_ADD]
  @p_Error_ID                        = NULL,
  @p_System                          = @v_System,
  @p_DB                              = @v_DB,
  @p_Obj                             = @v_Obj,
  @p_App                             = @v_App,
  @p_User                            = @v_User,
  @p_Spid                            = @v_Spid,
  @p_Description                     = NULL,
  @p_StartTime                       = @v_Now,
  @p_EndTime                         = NULL,
  @p_ActivityLog_ID                  = @v_ActivityLog_ID OUTPUT;     
 
-- Start the batch
IF @p_Batch_ID IS NULL
BEGIN  -- Create the batch
  SELECT @v_BatchStatus              = 1;
  EXEC [DBCATCH].[dbo].[PU_Batch_ADD]
    @p_Batch_ID                      = @p_Batch_ID OUTPUT
   ,@p_Type_ID                       = @v_Batch_Type_ID
   ,@p_Name                          = @v_Obj
   ,@p_Parent_ID                     = @v_ActivityLog_ID;
 
END;   -- Create the batch   
-- Add the next record to the batch
EXEC [DBCATCH].[dbo].[PU_BatchDetail_ADD]
  @p_Batch_ID                        = @p_Batch_ID,
  @p_Object_ID                       = @v_ActivityLog_ID,
  @p_Object                          = @v_Obj;           
  BEGIN TRY
 
    SELECT 'This is PU_Demo2'
  END TRY
 
  BEGIN CATCH
 
    SELECT @v_Error                  = 100000
          ,@v_MSG                    = CONVERT(NVARCHAR(2500), 
                                       'Error: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_NUMBER(), -1)) + 
                                       ' Severity: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_SEVERITY(), -1)) +
                                       ' State: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_STATE(), -1)) +
                                       ' Line: ' +  CONVERT([NVARCHAR](255), ISNULL(ERROR_LINE(), -1)) +
                                       ' Procedure: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_PROCEDURE(), @v_Obj)) +
                                       ' MSG: Error from [PU_Demo2]  with error:' + ISNULL(ERROR_MESSAGE(), ''));  
    EXEC [DBCATCH].[dbo].[PU_ErrorLog_ADD]
      @p_Error_ID                    = @v_Error,
      @p_System                      = @v_System,
      @p_DB                          = @v_DB,
      @p_Obj                         = @v_Obj,
      @p_App                         = @v_App,
      @p_User                        = @v_User,
      @p_ErrorMsg                    = @v_MSG;
 
    EXEC [DBCATCH].[dbo].[PU_ActivityLog_UPD]
      @p_ActivityLog_ID              = @v_ActivityLog_ID,
      @p_EndTime                     = @v_Now,
      @p_Error_ID                    = @v_Error,
      @p_Active                      = 0,
      @p_Description                 = @v_MSG,
      @p_DateModified                = @v_Now;    
    EXEC [DBCATCH].[dbo].[PU_BatchDetail_UPD]
      @p_ActivityLog_ID              = @v_ActivityLog_ID
     ,@p_Batch_ID                    = @p_Batch_ID
     ,@p_Now                         = @v_Now;         
    RETURN @v_Error;
 
  END CATCH;
EOF: 
/* Cleanup */
/* Update the activity Log */
EXEC [DBCATCH].[dbo].[PU_ActivityLog_UPD]
  @p_ActivityLog_ID                  = @v_ActivityLog_ID,
  @p_EndTime                         = @v_Now,
  @p_Active                          = 0,
  @p_DateModified                    = @v_Now;
 
/* Update the batch detail */
EXEC [DBCATCH].[dbo].[PU_BatchDetail_UPD]
  @p_ActivityLog_ID                  = @v_ActivityLog_ID
 ,@p_Batch_ID                        = @p_Batch_ID
 ,@p_Now                             = @v_Now;  
 
IF @v_BatchStatus                    = 1
BEGIN
  UPDATE [DBCATCH].[dbo].[TU_Batch]     WITH (ROWLOCK)
     SET [Active]                    = 0
	    ,[DateModified]              = GETDATE()
   WHERE [Batch_ID]                  = @p_Batch_ID;
END;  
GO   
IF OBJECT_ID('[dbo].[PU_Demo2]') IS NOT NULL
BEGIN
	PRINT '>>> CREATED PROCEDURE [dbo].[PU_Demo2] <<<';
END;
ELSE 
BEGIN
	PRINT '>>> FAILED TO CREATE PROCEDURE [dbo].[PU_Demo2] <<<';
END;
GO

Executing PU_Demo1

The last procedure (which is really the first procedure/wrapper) will call the other two procedures. To create PU_Demo1, copy the template to a new query window, press CTRL_SHIFT+M, and then replace ==>> Start Code Here! with the following commands:

SELECT 'This is PU_Demo1';
		EXEC PU_Demo2
		  @p_Batch_ID = @p_Batch_ID;
		EXEC PU_Demo3
		  @p_Batch_ID = @p_Batch_ID;

I will show the complete code in a moment below, but first look at the code snippet above. We will execute PU_Demo1 which will have a NULL @p_Batch_ID.

  1. It will then call PU_Batch_ADD which will insert a record for the Batch table
  2. Assigns a value to @p_Batch_ID which we will receive back in the PU_Demo1 scope as an output param
  3. Sets the @v_Batchstatus to 1
  4. Will call PU_BatchDetail_ADD that will insert a record in the batch detail table
  5. Call PU_Demo2
    1. Will call PU_BatchDetail_ADD that will insert a record in the batch detail table
    2. Call PU_BatchDetailUPD that will update the batchdetail record with the end date
  6. Call PU_Demo2
    1. Will call PU_BatchDetail_ADD that will insert a record in the batch detail table
    2. Call PU_BatchDetailUPD that will update the batchdetail record with the end date
  7. Call PU_BatchDetailUPD that will update the batchdetail record with the end date
  8. Update TU_Batch and close the batch with an end date.

Here is the code for PU_Demo1 shown below:

USE DBCATCH
GO
IF EXISTS(SELECT 1
            FROM [dbo].[sysobjects] (NOLOCK)
           WHERE [id]                                            = OBJECT_ID(N'[dbo].[PU_Demo1]')
             AND OBJECTPROPERTY([id], N'IsProcedure')            = 1) 
BEGIN 
  DROP PROCEDURE [dbo].[PU_Demo1];
 
  PRINT '>>> DROPPED PROCEDURE [dbo].[PU_Demo1] <<<';
END 
GO
CREATE PROC [dbo].[PU_Demo1]
  @p_Batch_ID                        [UNIQUEIDENTIFIER]          = NULL
WITH ENCRYPTION
AS
/*************************************************************************************************/ 
/* Name        : PU_Demo1                                                      */
/* Version     : 1.0                                                                             */
/* Author      : Jared Kirkpatrick                                                               */
/* Date        : 2019-05-24                                                                      */
/* Description :                                                                                 */
/*************************************************************************************************/ 
/* Date        : Version: Who: Description                                                       */
/*************************************************************************************************/ 
/* 2019-05-24  : 1.0    : JSK: Initial Release.                                                */
/*************************************************************************************************/ 
SET NOCOUNT ON;
/* Declare Variables */
DECLARE @v_Error                     [INT]
       ,@v_MSG                       [NVARCHAR](500)
       ,@v_DB                        [NVARCHAR](100)
       ,@v_Obj                       [NVARCHAR](100)
       ,@v_App                       [NVARCHAR](100)
       ,@v_User                      [NVARCHAR](100)
       ,@v_Spid                      [NVARCHAR](100)
       ,@v_ActivityLog_ID            [INT]
       ,@v_ADesc                     [NVARCHAR](512)
       ,@v_BatchStatus               [BIT]
       ,@v_Now                       [DATETIME]
       ,@v_LastErrorDate             [DATETIME]
       ,@v_SystemName                [NVARCHAR](100)
       ,@v_RowCount                  [INT]
       ,@v_Count                     [INT]
       ,@v_MinINT                    [INT]
       ,@v_MaxINT                    [INT]
       ,@v_Version                   [INT]       
       ,@v_KeyWord                   [SYSNAME]   
       ,@v_MSSQL_ID                  [INT]
       ,@v_Domain                    [NVARCHAR](100)  
       ,@v_Batch_Type_ID             [INT] 
	   ,@v_System                    [NVARCHAR](100)
       ,@v_SQL                       [NVARCHAR](4000)  
       ,@v_UserName                  [NVARCHAR](100)
       ,@v_PW                        [NVARCHAR](100)
       ,@v_Debug                     [INT];
 
SELECT @v_Now                        = GETDATE()
      ,@v_DB                         = DB_NAME()
      ,@v_Obj                        = '[DBCATCH].[dbo].[PU_Demo1]'
      ,@v_App                        = APP_NAME()
      ,@v_User                       = ISNULL(ORIGINAL_LOGIN(), USER_NAME()) 
      ,@v_Spid                       = CONVERT([NVARCHAR](25), @@SPID)
	  ,@v_Batch_Type_ID              = [dbo].[FSU_Code_GET]('Batch', 'PROCEDURE')
	  ,@v_System                     = HOST_NAME()
      ,@v_Debug                      = 0;  
 
 
/* Create Temp Tables */
 
IF @v_Debug                          = 1
BEGIN -- debug = 1 Ref: 1
  SELECT 'REF:1 - Define Static Variables';
  SELECT @v_Now                      AS [GETDATE]
        ,@v_DB                       AS [DB]
        ,@v_Obj                      AS [OBJ]
        ,@v_App                      AS [APP]
        ,@v_User                     AS [USER]
        ,@v_Spid                     AS [SPID]
        ,@p_Batch_ID                 AS [Batch_ID];
 
END; -- debug = 1    
 
/* Start the activity_log */
EXEC [DBCATCH].[dbo].[PU_ActivityLog_ADD]
  @p_Error_ID                        = NULL,
  @p_System                          = @v_System,
  @p_DB                              = @v_DB,
  @p_Obj                             = @v_Obj,
  @p_App                             = @v_App,
  @p_User                            = @v_User,
  @p_Spid                            = @v_Spid,
  @p_Description                     = NULL,
  @p_StartTime                       = @v_Now,
  @p_EndTime                         = NULL,
  @p_ActivityLog_ID                  = @v_ActivityLog_ID OUTPUT;     
 
-- Start the batch
IF @p_Batch_ID IS NULL
BEGIN  -- Create the batch
  SELECT @v_BatchStatus              = 1;
  EXEC [DBCATCH].[dbo].[PU_Batch_ADD]
    @p_Batch_ID                      = @p_Batch_ID OUTPUT
   ,@p_Type_ID                       = @v_Batch_Type_ID
   ,@p_Name                          = @v_Obj
   ,@p_Parent_ID                     = @v_ActivityLog_ID;
 
END;   -- Create the batch   
-- Add the next record to the batch
EXEC [DBCATCH].[dbo].[PU_BatchDetail_ADD]
  @p_Batch_ID                        = @p_Batch_ID,
  @p_Object_ID                       = @v_ActivityLog_ID,
  @p_Object                          = @v_Obj;           
  BEGIN TRY
 
        SELECT 'This is PU_Demo1';
		EXEC PU_Demo2
		  @p_Batch_ID = @p_Batch_ID;
		EXEC PU_Demo3
		  @p_Batch_ID = @p_Batch_ID;
  END TRY
 
  BEGIN CATCH
 
    SELECT @v_Error                  = 100000
          ,@v_MSG                    = CONVERT(NVARCHAR(2500), 
                                       'Error: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_NUMBER(), -1)) + 
                                       ' Severity: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_SEVERITY(), -1)) +
                                       ' State: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_STATE(), -1)) +
                                       ' Line: ' +  CONVERT([NVARCHAR](255), ISNULL(ERROR_LINE(), -1)) +
                                       ' Procedure: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_PROCEDURE(), @v_Obj)) +
                                       ' MSG: Error from [PU_Demo1]  with error:' + ISNULL(ERROR_MESSAGE(), ''));  
    EXEC [DBCATCH].[dbo].[PU_ErrorLog_ADD]
      @p_Error_ID                    = @v_Error,
      @p_System                      = @v_System,
      @p_DB                          = @v_DB,
      @p_Obj                         = @v_Obj,
      @p_App                         = @v_App,
      @p_User                        = @v_User,
      @p_ErrorMsg                    = @v_MSG;
 
    EXEC [DBCATCH].[dbo].[PU_ActivityLog_UPD]
      @p_ActivityLog_ID              = @v_ActivityLog_ID,
      @p_EndTime                     = @v_Now,
      @p_Error_ID                    = @v_Error,
      @p_Active                      = 0,
      @p_Description                 = @v_MSG,
      @p_DateModified                = @v_Now;    
    EXEC [DBCATCH].[dbo].[PU_BatchDetail_UPD]
      @p_ActivityLog_ID              = @v_ActivityLog_ID
     ,@p_Batch_ID                    = @p_Batch_ID
     ,@p_Now                         = @v_Now;         
    RETURN @v_Error;
 
  END CATCH;
EOF: 
/* Cleanup */
/* Update the activity Log */
EXEC [DBCATCH].[dbo].[PU_ActivityLog_UPD]
  @p_ActivityLog_ID                  = @v_ActivityLog_ID,
  @p_EndTime                         = @v_Now,
  @p_Active                          = 0,
  @p_DateModified                    = @v_Now;
 
/* Update the batch detail */
EXEC [DBCATCH].[dbo].[PU_BatchDetail_UPD]
  @p_ActivityLog_ID                  = @v_ActivityLog_ID
 ,@p_Batch_ID                        = @p_Batch_ID
 ,@p_Now                             = @v_Now;  
 
IF @v_BatchStatus                    = 1
BEGIN
  UPDATE [DBCATCH].[dbo].[TU_Batch]     WITH (ROWLOCK)
     SET [Active]                    = 0
	    ,[DateModified]              = GETDATE()
   WHERE [Batch_ID]                  = @p_Batch_ID;
END;  
GO   
IF OBJECT_ID('[dbo].[PU_Demo1]') IS NOT NULL
BEGIN
	PRINT '>>> CREATED PROCEDURE [dbo].[PU_Demo1] <<<';
END;
ELSE 
BEGIN
	PRINT '>>> FAILED TO CREATE PROCEDURE [dbo].[PU_Demo1] <<<';
END;
GO

Now we execute PU_Demo1. Open a new window and type:

EXEC PU_Demo1

Then hit ALT+X or click the execute button. Your results should look like the following:

you love our t-sql framework

So far, so good. The output is exactly what we expected. Now, look at the framework tables and validate the framework works as we expect it to.

First, check the TU_Errorlog and TU_ActivityLog tables:

I would normally get very excited here. We have one error that we coded for PU_Demo3 that showed up in the TU_Errorlog and in the activity log for object PU_Demo3. We already tested this functionality in the last article, so we will reserve the happy dance until we see the batching work.

Last, check the TU_Batch and TU_BatchDetail tables to validate our the last of our feature list:

Very Happy Outcomes

Now we are very happy. We have a record in the batch table and 1 in each of the corresponding batchdetail records grouped together with the batch_id. It shows us a nice ranking (which we could get just by ordering on the batchdetail_id groubed by the Batch_ID. It’s a nice elevator pitch that may get you some eye candy from management. The Object_ID will be the same activity_ID’s in the TU_activitylog table. We can see just how long each segment ran and we can see how long the batch took to run.

We have successfully created a functional framework with three key features. It would help any developer start rapid code development within this T-SQL framework. The first two features are a must-have for any piece of code that you want to take to an enterprise level. The batching of procedures is nice but is was picked as a feature for a few reasons:

  1. You can expand the feature set. Continue this development and create the next step, which would be to extend batching to also accommodate record set batching. Create a roadmap and write down your vision of how it should work and start developing. Go to it and have fun doing it!
  2. To show you that if you can dream it, you can build it. Don’t let my small introduction limit your T-SQL framework. You are limited only by your imagination.
  3. Because I thought it was a pretty cool feature, so I implemented it. You have a cool idea…go implement it and then write about it and share it.

I hope this series provided some insight into how you can make the day-to-day easier with some development. It also takes the load off the mundane, tedious tasks so that you can get to the good stuff — developing hard core logic?

The post The Finished Product: Framework, pt. 5 appeared first on SQL Solutions Group.

]]> https://sqlsolutionsgroup.com/t-sql-framework-2/feed/ 0 T-SQL Batching: Framework, pt. 4 https://sqlsolutionsgroup.com/t-sql-batching/ https://sqlsolutionsgroup.com/t-sql-batching/#respond Tue, 27 Aug 2019 07:38:11 +0000 http://sqlsolutionsgroup.com/?p=4239 In this fourth installment of the framework series, we will add a T-SQL batching feature to our existing framework we have developed in earlier installments of this series. This feature set is dependent on the work we have previously, done. If you have not been following along, please go back so that everything will work […]

The post T-SQL Batching: Framework, pt. 4 appeared first on SQL Solutions Group.

]]> In this fourth installment of the framework series, we will add a T-SQL batching feature to our existing framework we have developed in earlier installments of this series. This feature set is dependent on the work we have previously, done. If you have not been following along, please go back so that everything will work as anticipated (part 1, part 2, part 3).

There are no boundaries to what you can add to this or your own framework. I have chosen these three features (error log, activity log, batching) so I could illustrate examples. The possibilities are limitless, hindered only by your imagination. This next feature is multi-faceted, but we will focus on just the idea of T-SQL batching procedure calls to further extend logging.

We could develop this feature more to include batching of rows, batching parent and child structures so that segregation of domain aspects of the data model are defined, batching of similar lines of business… In short, we could develop batching out to almost any type of grouping one would need programmatically to have data-driven processes leveraged within the framework.

Creating T-SQL Batching: First Steps

As stated above, we will focus on developing the batching feature to extend the logging and information needed when we execute stored procedures as part of the framework. Even though we are focused on just this aspect of the feature, we will develop it in a way that it will scale if one would want to extend the feature’s functionality, which I will point out as we develop. There are a couple of moving parts for this feature (especially since we want it to easily expand in the future), so this will be a longer post. We will keep it easy to consume just as we have done with earlier posts.

Since we have some moving parts, let’s create a basic outline of precedence so that we follow the checklist as we develop. I highly recommend you take the time on any new project to get your thoughts down and have a road map with a clear vision of what you want to end up with as a solution.

  1. Create the Batch table (this is kind of like a header table)
  2. Create the batch detail table
  3. Add a “Batching” type code to our code group and code tables
  4. Create a scalar function to create new IDs for the Batch table
  5. Create the Insert/Update (DLL) procedures for the batching feature
  6. Update the Template (there will be some logic also involved past just proc calls)
  7. Remove all data from the Error log/Activity Log/Batching Tables
  8. Create a demo to test and validate our features
  9. Review the results
  10. Do a happy dance as customary when any development project is complete

Scale: Planning for the Future

There is a lot to do in developing this feature. The hard part is already done — we have defined above exactly what we need to do. Now we just need to create simple processes to complete each stage independently, and add the completed tasks to the collection like a puzzle until we are done … it really is that simple.

Since we may extend this feature in the future, we should allow for scalability, especially as we create the structures that will hold the data for this feature. Let’s explore what we envision this feature to do for us in the framework. We have an error handler, and we have a system for logging the details about every stored procedure we create and execute. Now we are developing a system that will group stored procedures that are nested, and define the details of this group like:

  1. What stored procedure started the chain of procedures being executed?
  2. How long did each segment of the batch (or each procedure) take?
  3. How long did the entire batch take?
  4. Did all procedures in the batch execute successfully?
  5. Which procedure (if any) did not complete successfully?

Some of the information gathered could be construed from other aspects of the framework. For example, what procedure did not execute successfully … we could find this easily from the activity log or even the error log. However, when you are looking at a specific process (especially a complicated process), looking for individual records that may or may not be affecting the process would not be as efficient as if you were looking at the stored procedures that specifically were called by the known process as a group.

Creating the Batch Table

The features we are creating also complement each other. In the last blog we observed that the work we developed in the error handling feature set was used in the activity log to enhance the logging capability of the activity log. Here, you’ll see that we leverage the features developed in the activity log to enhance the logging of the T-SQL batching process.

At this point we should have a pretty clear vision as to what we expect from our feature set; we have a clear road map to follow; and we have a lot of work to do so let’s get started!

The first task is to create the Batch table, which if you will remember constitutes as a header, so to speak, as listed below:

/*=======================================================================*/
/****** Object:  Table [dbo].[TU_Batch]     Script Date: 2019-08-21 ******/
/*=======================================================================*/
	PRINT CHAR(10) + CHAR(13) + 'Creating table [dbo].[TU_Batch].';	
 
 
CREATE TABLE [dbo].[TU_Batch]
(
  [Batch_ID]	[UNIQUEIDENTIFIER]   NOT NULL
    CONSTRAINT [PK_Batch_ID]   
    PRIMARY KEY CLUSTERED
    CONSTRAINT [DF_Batch_ID]
    DEFAULT CONVERT(UNIQUEIDENTIFIER, CONVERT(BINARY(10), NEWID()) + 
                                      CONVERT(BINARY(6), GETDATE())) 
 ,[Type_ID]     [INT]                NOT NULL
    CONSTRAINT [FK_TU_Batch_Type_ID]
       FOREIGN KEY ([Type_ID]) 
    REFERENCES [TU_Code]([Code_ID]) 
    CONSTRAINT [DF_TU_Batch_Type_ID]
    DEFAULT 0    
 ,[Name]        [NVARCHAR](100)     NOT NULL
    CONSTRAINT [DF_TU_Batch_Name]
    DEFAULT 'Batch:' + CONVERT(NVARCHAR(10), GETDATE(), 110)    
 ,[Parent_ID]   [INT]               NULL
 ,[Active]      [BIT]               NOT NULL
    CONSTRAINT [DF_TU_Batch_Active]
    DEFAULT 1
 ,[DateCreated] [DATETIME]          NOT NULL
    CONSTRAINT [DF_TU_Batch_DateCreated]
    DEFAULT GETDATE()
 ,[DateModified] [DATETIME]         NOT NULL
    CONSTRAINT [DF_TU_Batch_DateModified]
    DEFAULT GETDATE()
) ON [PRIMARY];
/* FK Index */
CREATE NONCLUSTERED INDEX idxfk_TU_Batch_Type_ID ON [TU_Batch]([Type_ID])

The Benefits of GUIDS

Before we continue, let’s look at our first column here: the batch_id that we have set as a uniqueidentifier. We also need to discuss the default that converts in and out of binary. Why would we go down this road if we are trying to keep things as simple as we can? There is no good reason for the feature set we are developing today, but we are building this feature set to scale. To this end we have some very good reasons as to why we want a uniqueidentifier:

  1. If and when we extend our batching feature, this feature may need to extend past processing in the database (or server for that matter), or origin. We can do batching through SSIS on a completely different server, so we should set the feature to be federated. Since identity uses integers that do not extend past the source database, the GUID is a practical data type that can be validated across the domain and even beyond.
  2. This is not an application, so we control creation and storage of the GUIDS. The GUIDS reads from right to left, so if we concatenate the random unique identifier with a time stamp on the end, we will have a sequential GUID that will not cause sporadic inserts across our extents.
  3. For future needs, we may want to copy a batch for reuse. Validating a copied GUID may be easier to identify than integers when not done programmatically.
[Batch_ID]	[UNIQUEIDENTIFIER]   NOT NULL
    CONSTRAINT [PK_Batch_ID]   
    PRIMARY KEY CLUSTERED
    CONSTRAINT [DF_Batch_ID]
    DEFAULT CONVERT(UNIQUEIDENTIFIER, CONVERT(BINARY(10), NEWID()) + 
                                      CONVERT(BINARY(6), GETDATE()))

Digging Deeper

This type of method is starting to go beyond level 1 development. Now that we have defined why you will see that, it is convenient to apply this type development to the feature we are building out. We started explaining the DEFAULT argument above in the second article of why we want a GUID, but as this is important, let’s expand on this before we move on:

  1. A GUID is 16 alpha-numeric characters read from right to left 
  2. The NEWID() Function is a server-side function that randomly creates a new 16-character UNIQUIDENTIFIER (nicknamed: GUID)
  3. The GETDATE() Function is a server-side function that returns a date
  4. We use the Binary datatype to concatenate the first 10 characters of the NEWD() with the first six characters of a sorted date — CONVERT(BINARY(10), NEWID()) + CONVERT(BINARY(6), GETDATE())
  5. Then we convert that to a UNIQUIDENTIFIER —

    CONVERT(

            UNIQUEIDENTIFIER,

    CONVERT(BINARY(10), NEWID()) + CONVERT(BINARY(6), GETDATE())

           )

Create Batch Detail Table

Next, we’ll create the batch detail table, shown below:

/*=========================================================================*/
/****** Object:  Table [dbo].[TU_BatchDetail]  Script Date: 2019-05-24    **/
/*=========================================================================*/
	PRINT CHAR(10) + CHAR(13) + 'Creating table [dbo].[TU_BatchDetail].';	
 
 
CREATE TABLE [dbo].[TU_BatchDetail]
(
  [BatchDetail_ID] [INT]  IDENTITY (1, 1)                                        NOT NULL 
    CONSTRAINT [PK_BatchDetail_ID]   
    PRIMARY KEY CLUSTERED
 ,[Batch_ID]       [UNIQUEIDENTIFIER]                                            NOT NULL
    CONSTRAINT [FK_TU_BatchDetail_Batch_ID]
       FOREIGN KEY ([Batch_ID]) 
    REFERENCES [TU_Batch]([Batch_ID]) 
 ,[BatchSize]      [INT]                                                         NOT NULL
    CONSTRAINT [DF_TU_Batch_BatchSize]
    DEFAULT 0     
 ,[OBJECT_ID]      [INT]                                                         NULL   
 ,[Entity_ID]      [UNIQUEIDENTIFIER]                                            NULL   
 ,[OBJECT]         [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[StartDate]      [DATETIME]                                                    NULL
 ,[EndDate]        [DATETIME]                                                    NULL
 ,[Rank]           [INT]                                                         NOT NULL
    CONSTRAINT [DF_TU_BatchDetail_Rank]
    DEFAULT 1
 ,[Active]          [BIT]                                                         NOT NULL
    CONSTRAINT [DF_TU_BatchDetail_Active]
    DEFAULT 1
 ,[DateCreated]     [DATETIME]                                                    NOT NULL
    CONSTRAINT [DF_TU_BatchDetail_DateCreated]
    DEFAULT GETDATE()
 ,[DateModified]    [DATETIME]                                                    NOT NULL
    CONSTRAINT [DF_TU_BatchDetail_DateModified]
    DEFAULT GETDATE()
) ON [PRIMARY];
/* FK Index */
CREATE NONCLUSTERED INDEX idxfk_TU_BatchDetail_Batch_ID ON [TU_BatchDetail]([Batch_ID])

 

This table is pretty standard, other than we put a couple of columns in that we will not need for this part of the feature, such as the batch size, which we would use to record size batches. That doesn’t mean this table won’t need modification when one decides to extend the batching features. But as you can see, we’re close enough that development could continue if desired.

Code Group and Code Tables

We’ve completed the first two items in our road map. Now we need to insert the type codes into our code group and code tables accordingly. In the second part of the framework, we created the Code Group and Code tables so that when we needed new codes we would not have to create new tables. Here we need new codes. We developed a great structure already so we can skip table creation because the structures we need are already available.

Let’s do this one at a time starting with the TU_CodeGroup table.

  • Look at what’s in the table currently:
T-sql batching framework
  • We see there is only one record for the Error handling feature, so let’s insert a record for batching:
SET IDENTITY_INSERT TU_CodeGroup ON;
INSERT [DBCATCH].[dbo].[TU_CodeGroup]
(      CodeGroup_ID, Name,                    [DESC])
SELECT 2,            'BATCH',                 'TYPE OF BATCHS'
  SET IDENTITY_INSERT TU_CodeGroup OFF;
  • Check the table again:
batching

Exactly what we were anticipating. Follow the same three steps and insert the codes into the code table for the batching types:

  • Look at what is in the table currently:
  • After looking at the data we insert the data we need for batching:
SET IDENTITY_INSERT TU_Code ON;
INSERT [DBCATCH].[dbo].[TU_Code]
      Code_ID, CodeGroup_ID, Name,                    [DESC])
SELECT 200,     2,            'DEFAULT',               'DEFAULT BATCH TYPE'
UNION
SELECT 201,     2,            'PROCEDURE',             'SP BATCH FOR ORDINAL COLLECTION'
UNION
SELECT 202,     2,            'RECORDCOUNT',           'SET NUMBER OF RECORDS TO be BATCHED'
  SET IDENTITY_INSERT TU_Code OFF;
  • Check the code tables again:

The Scalar Function

Now that we have successfully entered our type codes, we move forward to creating a scalar function. This function does exactly as planned with the table definition: it will get a new uniqueidentifier for us.

We do this through a scalar function. It’s easier to create one into a variable and use that through nested operation versus a lookup each time. If we use a scalar function each time, why go to the trouble of creating a default in the table definition? That’s for completeness and for our protection. Instead of erroring out of a process, if a null is ever passed it will get an ID and allow the process to continue. With that, here’s the code for the scalar function as we continue our development:

USE [DBCATCH];
GO
IF EXISTS(SELECT 1
            FROM [INFORMATION_SCHEMA].[ROUTINES] (NOLOCK)
           WHERE [SPECIFIC_SCHEMA]   = 'dbo'
             AND [SPECIFIC_NAME]     = 'FSU_GetGUID')
BEGIN
  DROP FUNCTION [dbo].[FSU_GetGUID];
  PRINT '<<< DROPPED FUNCTION [dbo].[FSU_GetGUID] >>>';
END;
GO
CREATE FUNCTION [dbo].[FSU_GetGUID] 
(
  @p_Guid                            [UNIQUEIDENTIFIER]
 ,@p_Date                            [DATETIME]	
)
RETURNS [UNIQUEIDENTIFIER]
WITH ENCRYPTION
AS
BEGIN -- Begin function
/************************************************************************/
/* Name         : FSU_GetGUID                                           */
/* Version	  : 1.0                                                   */
/* Author	  : Jared Kirkpatrick                                     */
/* Date	  : 2019-08-24                                            */
/* Description  : Gets a new binary sorted guid                         */
/************************************************************************/
/* Date         : Version: Who: Description                             */
/************************************************************************/
/* 2019-08-24   : 1.0    : JSK: Initial Release.                      */
/************************************************************************/
/* Declare Variables */
  DECLARE @v_GUID                    [UNIQUEIDENTIFIER];
 
  SELECT @v_GUID                     = CONVERT(UNIQUEIDENTIFIER, 
                                       CONVERT(BINARY(10), @p_Guid) + 
                                       CONVERT(BINARY(6), @p_Date)
                                              ) 
  RETURN @v_GUID
 
END;  -- Begin function
GO
IF OBJECT_ID('[dbo].[FSU_GetGUID]') IS NOT NULL
BEGIN
	PRINT '<<< CREATED FUNCTION [dbo].[FSU_GetGUID] >>>';
END;
ELSE 
BEGIN
	PRINT '<<< FAILED TO CREATE FUNCTION [dbo].[FSU_GetGUID] >>>';
END;
GO

Creating the Procedures

Next on the list is to create the Insert/Update procedures for the Batch and Batch Detail tables. We should not have any reason to modify the header records. The Batch table will only need an Insert procedure, whereas the Detail tables will need both an Insert and an update procedure.

USE DBCATCH
GO
IF EXISTS(SELECT 1
            FROM [dbo].[sysobjects] (NOLOCK)
           WHERE [id]                                            = OBJECT_ID(N'[dbo].[PU_Batch_ADD]')
             AND OBJECTPROPERTY([id], N'IsProcedure')            = 1) 
BEGIN 
  DROP PROCEDURE [dbo].[PU_Batch_ADD];
 
  PRINT '<<< DROPPED PROCEDURE [dbo].[PU_Batch_ADD] >>>';
END 
GO
CREATE PROC [dbo].[PU_Batch_ADD]
  @p_Type_ID                         [INT]                       = 700,
  @p_Name                            [NVARCHAR](100)             = NULL,
  @p_Parent_ID                       [INT]                       = NULL,
  @p_Batch_ID                        [UNIQUEIDENTIFIER]          = NULL OUTPUT
WITH ENCRYPTION
AS  
/*************************************************************************************************/ 
/* Name        : PU_Batch_ADD                                                                    */
/* Version     : 1.0                                                                             */
/* Author      : Jared Kirkpatrick                                                               */
/* Date        : 2019-08-24                                                                      */
/* Description : Insertes a record in the Batch Header                                           */
/*************************************************************************************************/ 
/* Date        : Version: Who: Description                                                       */
/*************************************************************************************************/ 
/* 2019-08-24  : 1.0    : REB: Initial Release.                                                */
/*************************************************************************************************/ 
SET NOCOUNT ON;
/* Declare Variables */
DECLARE @v_Now                       [DATETIME]
       ,@v_Domain                    [NVARCHAR](100)
       ,@v_IPAddress                 [NVARCHAR](100)
       ,@v_SQL                       [NVARCHAR](4000)
       ,@v_UserName                  [NVARCHAR](100)
       ,@v_PW                        [NVARCHAR](100)
       ,@v_Error                     [INT]
       ,@v_ActivityLog_ID            [INT]
       ,@v_DB                        [NVARCHAR](100)
       ,@v_Obj                       [NVARCHAR](100)
       ,@v_App                       [NVARCHAR](100)
       ,@v_User                      [NVARCHAR](100)            
       ,@v_Spid                      [NVARCHAR](100)
       ,@v_Debug                     [BIT]       
       ,@v_MSG                       [NVARCHAR](1000)
       ,@v_SystemName                [NVARCHAR](100)
       ,@v_Connect_Type_ID           [INT]
       ,@v_RowCount                  [INT]
       ,@v_Count                     [INT]
       ,@v_Version                   [INT]
       ,@v_MSSQL_ID                  [INT];
 
/* Define Variables */
SELECT @v_Now                        = GETDATE()
      ,@v_DB                         = DB_NAME()
      ,@v_Obj                        = '[DBCATCH].[dbo].[PU_Batch_ADD]'
      ,@v_App                        = APP_NAME()
      ,@v_User                       = ISNULL(ORIGINAL_LOGIN(), USER_NAME()) 
      ,@v_Spid                       = CONVERT([NVARCHAR](25), @@SPID) 
      ,@v_Debug                      = 0;
 
IF @p_Name IS NULL
BEGIN
  SELECT @p_Name                     = 'Batch:' + CONVERT(NVARCHAR(10), @v_Now, 112);      
END;         
 
IF @p_Batch_ID IS NULL
BEGIN
  SELECT @p_Batch_ID                 = CONVERT(UNIQUEIDENTIFIER, CONVERT(BINARY(10), NEWID()) + 
                                                 CONVERT(BINARY(6), GETDATE()));      
END;                                                 
 
IF @v_Debug                          = 1
BEGIN -- debug = 1 Ref: 1
  SELECT 'REF:1 - Define Static Variables';
  SELECT @v_Now                      AS [GETDATE]
        ,@v_DB                       AS [DB]
        ,@v_Obj                      AS [OBJ]
        ,@v_App                      AS [APP]
        ,@v_User                     AS [USER]
        ,@v_Spid                     AS [SPID]
        ,@p_Batch_ID                 AS [Batch_ID];
 
END; -- debug = 1         
BEGIN TRANSACTION 
  BEGIN TRY
    INSERT [DBCATCH].[dbo].[TU_Batch]
    (
      [Batch_ID],
      [Type_ID],
      [Name],
      [Parent_ID]
    )
    VALUES
    (
      @p_Batch_ID,
      @p_Type_ID,
      @p_Name,
      @p_Parent_ID
    )
  END TRY
 
  BEGIN CATCH
 
    SELECT @v_Error                  = 100001
          ,@v_MSG                    = CONVERT(NVARCHAR(2500), 
                                       'Error: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_NUMBER(), -1)) + 
                                       ' Severity: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_SEVERITY(), -1)) +
                                       ' State: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_STATE(), -1)) +
                                       ' Line: ' +  CONVERT([NVARCHAR](255), ISNULL(ERROR_LINE(), -1)) +
                                       ' Procedure: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_PROCEDURE(), @v_Obj)) +
                                       ' MSG: Error Inserting into [DBCATCH].[dbo].[TU_Batch] with error:' + ISNULL(ERROR_MESSAGE(), ''));  
    EXEC [DBCATCH].[dbo].[PU_ErrorLog_ADD]
      @p_Error_ID                    = @v_Error,
      @p_System                      = NULL,
      @p_DB                          = @v_DB,
      @p_Obj                         = @v_Obj,
      @p_App                         = @v_App,
      @p_User                        = @v_User,
      @p_ErrorMsg                    = @v_MSG;
 
    ROLLBACK TRAN;
    RETURN @v_Error;
 
  END CATCH;
COMMIT TRANSACTION;
GO
IF OBJECT_ID('[dbo].[PU_Batch_ADD]') IS NOT NULL
BEGIN
	PRINT '<<< CREATED PROCEDURE [dbo].[PU_Batch_ADD] >>>';
END;
ELSE 
BEGIN
	PRINT '<<< FAILED TO CREATE PROCEDURE [dbo].[PU_Batch_ADD] >>>';
END;
GO

That takes care of the insert procedure for the Batch table. Notice we customized the TRY CATCH error message. The Insert Procedure for the Batch Detail record is below:

USE DBCATCH
GO
IF EXISTS(SELECT 1
            FROM [dbo].[sysobjects] (NOLOCK)
           WHERE [id]                                            = OBJECT_ID(N'[dbo].[PU_BatchDetail_ADD]')
             AND OBJECTPROPERTY([id], N'IsProcedure')            = 1) 
BEGIN 
  DROP PROCEDURE [dbo].[PU_BatchDetail_ADD];
 
  PRINT '<<< DROPPED PROCEDURE [dbo].[PU_BatchDetail_ADD] >>>';
END; 
GO
CREATE PROC [dbo].[PU_BatchDetail_ADD]
  @p_Batch_ID                        [UNIQUEIDENTIFIER],
  @p_BatchSize                       [INT]                       = 0,
  @p_Object_ID                       [INT]                       = NULL,
  @p_Entity_ID                       [UNIQUEIDENTIFIER]          = NULL,
  @p_Object                          [NVARCHAR](100)             = NULL,
  @p_StartDate                       [DATETIME]                  = NULL,
  @p_EndDate                         [DATETIME]                  = NULL,
  @p_Rank                            [INT]                       = NULL
WITH ENCRYPTION
AS
/*************************************************************************************************/ 
/* Name        : PU_BatchDetail_ADD                                                              */
/* Version     : 1.0                                                                             */
/* Author      : Jared Kirkpatrick                                                               */
/* Date        : 2019-08-24                                                                      */
/* Description : Inserts a record for the batch detail                                           */
/*************************************************************************************************/ 
/* Date        : Version: Who: Description                                                       */
/*************************************************************************************************/ 
/* 2019-08-24  : 1.0    : REB: Initial Release.                                                */
/*************************************************************************************************/
SET NOCOUNT ON;
/* Declare Variables */
DECLARE @v_Now                       [DATETIME]
       ,@v_Domain                    [NVARCHAR](100)
       ,@v_IPAddress                 [NVARCHAR](100)
       ,@v_SQL                       [NVARCHAR](4000)
       ,@v_UserName                  [NVARCHAR](100)
       ,@v_PW                        [NVARCHAR](100)
       ,@v_Error                     [INT]
       ,@v_ActivityLog_ID            [INT]
       ,@v_DB                        [NVARCHAR](100)
       ,@v_Obj                       [NVARCHAR](100)
       ,@v_App                       [NVARCHAR](100)
       ,@v_User                      [NVARCHAR](100)            
       ,@v_Spid                      [NVARCHAR](100)
       ,@v_Debug                     [BIT]       
       ,@v_MSG                       [NVARCHAR](1000)
       ,@v_SystemName                [NVARCHAR](100)
       ,@v_Connect_Type_ID           [INT]
       ,@v_RowCount                  [INT]
       ,@v_Count                     [INT]
       ,@v_Version                   [INT]
       ,@v_MSSQL_ID                  [INT];
 
/* Define Variables */
SELECT @v_Now                        = GETDATE()
      ,@v_DB                         = DB_NAME()
      ,@v_Obj                        = '[DBCATCH].[dbo].[PU_BatchDetail_ADD]'
      ,@v_App                        = APP_NAME()
      ,@v_User                       = ISNULL(ORIGINAL_LOGIN(), USER_NAME()) 
      ,@v_Spid                       = CONVERT([NVARCHAR](25), @@SPID)
      ,@v_Debug                      = 0;
 
IF @p_Rank                           IS NULL
BEGIN
  SELECT @p_Rank                     = ISNULL(MAX([Rank]) + 1, 1)
    FROM [DBCATCH].[dbo].[TU_BatchDetail] (NOLOCK)
   WHERE [Batch_ID]                  = @p_Batch_ID; 
END;  
IF @p_StartDate                      IS NULL
BEGIN
  SELECT @p_StartDate                = @v_Now;
END;  
IF @v_Debug = 1
BEGIN -- debug = 1 Ref: 1
  SELECT 'REF:1 - Define Static Variables';
  SELECT @v_Now                      AS [GETDATE]
        ,@v_DB                       AS [DB]
        ,@v_Obj                      AS [OBJ]
        ,@v_App                      AS [APP]
        ,@v_User                     AS [USER]
        ,@v_Spid                     AS [SPID]
        ,@p_Batch_ID                 AS [Batch_ID];
 
END; -- debug = 1   
BEGIN TRANSACTION 
  BEGIN TRY
    INSERT [DBCATCH].[dbo].[TU_BatchDetail]
    (
      [Batch_ID],
      [BatchSize],
      [OBJECT_ID],
      [Entity_ID],
      [OBJECT],
      [StartDate],
      [EndDate],
      [Rank]
    )
    VALUES
    (
      @p_Batch_ID,
      @p_BatchSize,
      @p_Object_ID,
      @p_Entity_ID,
      @p_Object,
      @p_StartDate,
      @p_EndDate,
      @p_Rank
    );
  END TRY
 
  BEGIN CATCH
 
    SELECT @v_Error                  = 100001
          ,@v_MSG                    = CONVERT(NVARCHAR(2500), 
                                        'Error: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_NUMBER(), -1)) + 
                                        ' Severity: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_SEVERITY(), -1)) +
                                        ' State: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_STATE(), -1)) +
                                        ' Line: ' +  CONVERT([NVARCHAR](255), ISNULL(ERROR_LINE(), -1)) +
                                        ' Procedure: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_PROCEDURE(), @v_Obj)) +
                                        ' MSG: Error Inserting into [DBCATCH].[dbo].[TU_Batch] with error:' + ISNULL(ERROR_MESSAGE(), ''));  
    EXEC [DBCATCH].[dbo].[PU_ErrorLog_ADD]
      @p_Error_ID                    = @v_Error,
      @p_System                      = NULL,
      @p_DB                          = @v_DB,
      @p_Obj                         = @v_Obj,
      @p_App                         = @v_App,
      @p_User                        = @v_User,
      @p_ErrorMsg                    = @v_MSG;
 
    ROLLBACK TRAN;
    RETURN @v_Error;
 
  END CATCH;
COMMIT TRANSACTION;
GO
IF OBJECT_ID('[dbo].[PU_BatchDetail_ADD]') IS NOT NULL
BEGIN
	PRINT '<<< CREATED PROCEDURE [dbo].[PU_BatchDetail_ADD] >>>';
END;
ELSE 
BEGIN
	PRINT '<<< FAILED TO CREATE PROCEDURE [dbo].[PU_BatchDetail_ADD] >>>';
END;
GO

Final Step: Update Procedure & Stop Times

The last task in your T-SQL batching is to create an update procedure for the batch detail records. This adds stop times for each nested iteration and for the batch itself.

USE DBCATCH
GO
IF EXISTS(SELECT 1
            FROM [dbo].[sysobjects] (NOLOCK)
           WHERE [id]                                            = OBJECT_ID(N'[dbo].[PU_BatchDetail_UPD]')
             AND OBJECTPROPERTY([id], N'IsProcedure')            = 1) 
BEGIN 
  DROP PROCEDURE [dbo].[PU_BatchDetail_UPD];
 
  PRINT '<<< DROPPED PROCEDURE [dbo].[PU_BatchDetail_UPD] >>>';
END 
GO
CREATE PROC [dbo].[PU_BatchDetail_UPD]
  @p_ActivityLog_ID              [BIGINT]
 ,@p_Batch_ID                    [UNIQUEIDENTIFIER]              = NULL
 ,@p_Now                         [DATETIME]                      = NULL
WITH ENCRYPTION
AS
/*************************************************************************************************/ 
/* Name        : PU_BatchDetail_UPD                                                              */
/* Version     : 1.0                                                                             */
/* Author      : Jared Kirkpatrick                                                               */
/* Date        : 2019-08-24                                                                      */
/* Description : Updating the Batch detail log                                                   */
/*************************************************************************************************/ 
/* Date        : Version: Who: Description                                                       */
/*************************************************************************************************/ 
/* 2019-08-24  : 1.0    : REB: Initial Release.                                                */
/*************************************************************************************************/
SET NOCOUNT ON;
/* Declare Variables */
/* Declare Variables */
DECLARE @v_Now                       [DATETIME]
       ,@v_Domain                    [NVARCHAR](100)
       ,@v_IPAddress                 [NVARCHAR](100)
       ,@v_SQL                       [NVARCHAR](4000)
       ,@v_UserName                  [NVARCHAR](100)
       ,@v_PW                        [NVARCHAR](100)
       ,@v_Error                     [INT]
       ,@v_ActivityLog_ID            [INT]
       ,@v_DB                        [NVARCHAR](100)
       ,@v_Obj                       [NVARCHAR](100)
       ,@v_App                       [NVARCHAR](100)
       ,@v_User                      [NVARCHAR](100)            
       ,@v_Spid                      [NVARCHAR](100)
       ,@v_Debug                     [BIT]       
       ,@v_MSG                       [NVARCHAR](1000)
       ,@v_SystemName                [NVARCHAR](100)
       ,@v_Connect_Type_ID           [INT]
       ,@v_RowCount                  [INT]
       ,@v_Count                     [INT]
       ,@v_Version                   [INT]
       ,@v_MSSQL_ID                  [INT];
 
/* Define Variables */
SELECT @v_Now                        = GETDATE()
      ,@v_DB                         = DB_NAME()
      ,@v_Obj                        = '[DBCATCH].[dbo].[PU_BatchDetail_UPD]'
      ,@v_App                        = APP_NAME()
      ,@v_User                       = ISNULL(ORIGINAL_LOGIN(), USER_NAME()) 
      ,@v_Spid                       = CONVERT([NVARCHAR](25), @@SPID) 
      ,@v_Debug                      = 0;
 
IF @v_Debug = 1
BEGIN -- debug = 1 Ref: 1
  SELECT 'REF:1 - Define Static Variables';
  SELECT @v_Now                      AS [GETDATE]
        ,@v_DB                       AS [DB]
        ,@v_Obj                      AS [OBJ]
        ,@v_App                      AS [APP]
        ,@v_User                     AS [USER]
        ,@v_Spid                     AS [SPID]
        ,@p_Batch_ID                 AS [Batch_ID];
 
END; -- debug = 1       
BEGIN TRANSACTION 
  BEGIN TRY
    UPDATE [DBCATCH].[dbo].[TU_BatchDetail]                         WITH (ROWLOCK)
       SET [EndDate]                 = @p_Now
          ,[Active]                  = 0
     WHERE [OBJECT_ID]               = @p_ActivityLog_ID
       AND [Batch_ID]                = @p_Batch_ID;
  END TRY
 
  BEGIN CATCH
 
    SELECT @v_Error                  = 100002
          ,@v_MSG                    = CONVERT(NVARCHAR(2500), 
                                       'Error: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_NUMBER(), -1)) + 
                                       ' Severity: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_SEVERITY(), -1)) +
                                       ' State: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_STATE(), -1)) +
                                       ' Line: ' +  CONVERT([NVARCHAR](255), ISNULL(ERROR_LINE(), -1)) +
                                       ' Procedure: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_PROCEDURE(), @v_Obj)) +
                                       ' MSG: Error Updating [DBCATCH].[dbo].[TU_BatchDetail] with error:' + ISNULL(ERROR_MESSAGE(), ''));  
    EXEC [DBCATCH].[dbo].[PU_ErrorLog_ADD]
      @p_Error_ID                    = @v_Error,
      @p_System                      = NULL,
      @p_DB                          = @v_DB,
      @p_Obj                         = @v_Obj,
      @p_App                         = @v_App,
      @p_User                        = @v_User,
      @p_ErrorMsg                    = @v_MSG;
 
    ROLLBACK TRAN;
    RETURN @v_Error;
 
  END CATCH;
COMMIT TRANSACTION;
GO
IF OBJECT_ID('[dbo].[PU_BatchDetail_UPD]') IS NOT NULL
BEGIN
	PRINT '<<< CREATED PROCEDURE [dbo].[PU_BatchDetail_UPD] >>>';
END;
ELSE 
BEGIN
	PRINT '<<< FAILED TO CREATE PROCEDURE [dbo].[PU_BatchDetail_UPD] >>>';
END;
GO

That completes the objects needed to make our T-SQL batching framework functional, and completes part 4 of the series. In part 5, the final installment, we’ll go over these batching features and wrap it all up.

GO TO PART FIVE

The post T-SQL Batching: Framework, pt. 4 appeared first on SQL Solutions Group.

]]> https://sqlsolutionsgroup.com/t-sql-batching/feed/ 0 The T-SQL Activity Log: Framework, pt. 3 https://sqlsolutionsgroup.com/the-tsql-activity-log/ https://sqlsolutionsgroup.com/the-tsql-activity-log/#respond Wed, 21 Aug 2019 12:34:21 +0000 http://sqlsolutionsgroup.com/?p=4215 The post The T-SQL Activity Log: Framework, pt. 3 appeared first on SQL Solutions Group.

]]> This segment of the framework builds onto what we have already created in Part 2 of the T-SQL Framework. That post showed you how to build the Error Engine, or build in tasks to do our error handling for us. In building the T-SQL Activity Log, we will leverage some of the structures and processes we have already built and tested. If you have not gone through and created the initial T-SQL Framework from part two, please go back and do that now.

The Activity Log will log every routine we create. This includes logging:

  • The start of the routine
  • The name of the routine
  • Who initiated the process
  • Where the process was started
  • When the process started
  • Error codes if any that the process encountered (this will be from our error engine we developed in part 2)
  • When the routine completed so we will have a baseline at least in time of how long it takes the process to run

Create the T-SQL Activity Log Table

Let’s jump right in and create the table we will need in order to build the T-SQL activity log information, shown in code:

/*============================================================================*/
/****** Object:  Table [dbo].[TU_ActivityLog]         Script Date: 2019-10-15                ******/
/*============================================================================*/
	PRINT CHAR(10) + CHAR(13) + 'Creating table [dbo].[TU_ActivityLog].';	
 
CREATE TABLE [dbo].[TU_ActivityLog] 
(
  [ActivityLog_ID] [BIGINT]  IDENTITY (1, 1)                                     NOT NULL
    CONSTRAINT [PK_ActivityLog_ID]   
    PRIMARY KEY CLUSTERED  
 ,[System]         [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[Error_ID]       [INT]                                                         NULL
    CONSTRAINT [FK_TU_ActivityLog_Error_ID]
       FOREIGN KEY ([Error_ID]) 
    REFERENCES [TU_Error]([Error_ID])   
 ,[DB]             [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[Obj]            [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[App]            [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[USER]           [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL 
 ,[SPID]           [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL 
 ,[Description]    [NVARCHAR](512)	COLLATE SQL_Latin1_General_CP1_CI_AS           NULL 
 ,[StartTime]      [DATETIME]                                                    NULL
 ,[EndTime]        [DATETIME]                                                    NULL 
 ,[Active]         [BIT]                                                         NOT NULL
    CONSTRAINT [DF_ActivityLog_Active]
    DEFAULT (1)
 ,[DateCreated]    [DATETIME]							     NOT NULL
    CONSTRAINT [DF_ActivityLog_DateCreated]
    DEFAULT GETDATE()
 ,[DateModified]    [DATETIME]			    			          NOT NULL
    CONSTRAINT [DF_ActivityLog_DateModified]
    DEFAULT GETDATE()			
) ON [PRIMARY];
 
 
/* FK Index */
CREATE NONCLUSTERED INDEX idxfk_TU_ActivityLog_Error_ID ON [TU_ActivityLog]([Error_ID])

Looking at the table definition, you can see we have several columns that were variables in the original framework when we created the error engine. We used most of those variables to define our customized error if we went into the begin catch part of the procedure orchestrated through the template we developed. The point is that we can use the variables outside of the scope of just the error handling.

We are reusing variables that we defined in the framework to facilitate information between features but doing so in a way that all roads lead back to Rome. When you use this methodology, you will find it makes your code simpler and easier for others to read. Most importantly, if we have developed and tested variables from one feature that we need in another, we don’t have to retest if we did our job correctly the first time. That cuts down on development/troubleshooting time as well. Enough about theory and methodology, let’s get back to some development!

Create Two Procedures

The next thing we need are two procedures: one to insert into the activity log, and another to update the activity log when we are done with the process — either at a successful completion or at the end of our Begin Catch if there was an error. The first stored procedure is the Insert, shown below:

USE DBCATCH
GO
 
IF EXISTS(SELECT 1
            FROM [dbo].[sysobjects] (NOLOCK)
           WHERE [id]                                            = OBJECT_ID(N'[dbo].[PU_ActivityLog_ADD]')
             AND OBJECTPROPERTY([id], N'IsProcedure')            = 1) 
 
BEGIN 
  DROP PROCEDURE [dbo].[PU_ActivityLog_ADD];
 
  PRINT '<<< DROPPED PROCEDURE [dbo].[PU_ActivityLog_ADD] >>>';
END 
GO
 
CREATE PROC [dbo].[PU_ActivityLog_ADD]
  @p_System                          [NVARCHAR](100)             = NULL
 ,@p_Error_ID                        [INT]                       = NULL
 ,@p_DB                              [NVARCHAR](100)             = NULL
 ,@p_Obj                             [NVARCHAR](100)             = NULL
 ,@p_App                             [NVARCHAR](100)             = NULL
 ,@p_User                            [NVARCHAR](100)             = NULL
 ,@p_SPID                            [NVARCHAR](100)             = NULL
 ,@p_Description                     [NVARCHAR](512)             = NULL
 ,@p_StartTime                       [DATETIME]                  = NULL
 ,@p_EndTime                         [DATETIME]                  = NULL
 ,@p_ActivityLog_ID                  [BIGINT]                       OUTPUT
WITH ENCRYPTION
AS
 
/*************************************************************************************************/ 
/* Name        : PU_ActivityLog_ADD                                                              */
/* Version     : 1.0                                                                             */
/* Author      : Jared Kirkpatrick                                                               */
/* Date        : 2019-10-15                                                                      */
/* Description : Adds a record to the Activity Log                                               */
/*************************************************************************************************/ 
/* Date        : Version: Who: Description                                                       */
/*************************************************************************************************/ 
/* 2019-10-15  : 1.0    : REB: Initial Release.                                                 */
/*************************************************************************************************/
 
SET NOCOUNT ON
 
/* Declare Variables */
DECLARE @v_Error                     [INT]
       ,@v_MSG                       [NVARCHAR](2500);
 
BEGIN TRANSACTION 
 
  BEGIN TRY
 
    INSERT [DBCATCH].[dbo].[TU_ActivityLog]
    (
      [System],
      [Error_ID],
      [DB],
      [Obj],
      [App],
      [USER],
      [SPID],
      [Description],
      [StartTime],
      [EndTime]
    )
    VALUES
    (
      @p_System,
      @p_Error_ID,
      @p_DB,
      @p_Obj,
      @p_App,
      @p_User,
      @p_SPID,
      @p_Description,
      @p_StartTime,
      @p_EndTime
    )
 
    SELECT @p_ActivityLog_ID         = @@IDENTITY;  
 
  END TRY
 
  BEGIN CATCH
 
    SELECT @p_Obj                    = '[DBCATCH].[dbo].[TU_ActivityLog]';
 
    SELECT @v_Error  = 100001
          ,@v_MSG    = CONVERT(NVARCHAR(2500), 
                     'Error: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_NUMBER(), -1)) + 
                     ' Severity: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_SEVERITY(), -1)) +
                     ' State: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_STATE(), -1)) +
                     ' Line: ' +  CONVERT([NVARCHAR](255), ISNULL(ERROR_LINE(), -1)) +
                     ' Procedure: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_PROCEDURE(), @p_Obj)) +
                     ' MSG: Error Inserting into [DBCATCH].[dbo].[TU_ActivityLog] with error:' + ISNULL(ERROR_MESSAGE(), ''));  
 
    EXEC [DBCATCH].[dbo].[PU_ErrorLog_ADD]
      @p_Error_ID                    = @v_Error,
      @p_System                      = @p_System,
      @p_DB                          = @p_DB,
      @p_Obj                         = @p_Obj,
      @p_App                         = @p_App,
      @p_User                        = @p_User,
      @p_ErrorMsg                    = @v_MSG;
 
    ROLLBACK TRAN;
    RETURN @v_Error;
 
  END CATCH;
 
COMMIT TRANSACTION;
GO
 
IF OBJECT_ID('[dbo].[PU_ActivityLog_ADD]') IS NOT NULL
BEGIN
	PRINT '<<< CREATED PROCEDURE [dbo].[PU_ActivityLog_ADD] >>>';
END;
ELSE 
BEGIN
	PRINT '<<< FAILED TO CREATE PROCEDURE [dbo].[PU_ActivityLog_ADD] >>>';
END;
GO

Take a closer look at the error and error message in the “begin try” part of the procedure.Building the T-SQL Activity Log

 

As you can see, we have changed the error number to match what we defined as our errors in TU_Error, as this error deals with an Insert. We also customized the error message to show it came while inserting into the activity log. The rest of the error will capture the server’s definition but this way we have set up our visuals where, if we see this in the error log, we will know where to look for answers.

We will do this with the update as well. Every time the code runs, it is running with a customized error handler to the routine at hand. Just a few seconds of time makes this a truly powerful tool, not only in development but through the lifecycle of the software process running on the server.

The Update Procedure

Next is the update procedure:

USE DBCATCH
GO
 
IF EXISTS(SELECT 1
            FROM [dbo].[sysobjects] (NOLOCK)
           WHERE [id]                                            = OBJECT_ID(N'[dbo].[PU_ActivityLog_UPD]')
             AND OBJECTPROPERTY([id], N'IsProcedure')            = 1) 
 
BEGIN 
  DROP PROCEDURE [dbo].[PU_ActivityLog_UPD];
 
  PRINT '<<< DROPPED PROCEDURE [dbo].[PU_ActivityLog_UPD] >>>';
END 
GO
 
CREATE PROC [dbo].[PU_ActivityLog_UPD]
  @p_ActivityLog_ID                  [BIGINT],
  @p_System                          [NVARCHAR](100)             = NULL,
  @p_Error_ID                        [INT]                       = NULL,
  @p_DB                              [NVARCHAR](100)             = NULL,
  @p_Obj                             [NVARCHAR](100)             = NULL,
  @p_App                             [NVARCHAR](100)             = NULL,
  @p_User                            [NVARCHAR](100)             = NULL,
  @p_Description                     [NVARCHAR](512)             = NULL,
  @p_StartTime                       [DATETIME]                  = NULL,
  @p_EndTime                         [DATETIME]                  = NULL,
  @p_Active                          [BIT]                       = 0,
  @p_DateModified                    [DATETIME]
WITH ENCRYPTION
AS
 
/*************************************************************************************************/ 
/* Name        : PU_ActivityLog_UPD                                                              */
/* Version     : 1.0                                                                             */
/* Author      : Jared Kirkpatrick                                                               */
/* Date        : 2019-05-24                                                                      */
/* Description : Updating the activity log                                                       */
/*************************************************************************************************/ 
/* Date        : Version: Who: Description                                                       */
/*************************************************************************************************/ 
/* 2019-05-24  : 1.0    : REB: Initial Release.                                                 */
/*************************************************************************************************/
 
SET NOCOUNT ON;
 
/* Declare Variables */
DECLARE @v_Error                     [INT]
       ,@v_MSG                       [NVARCHAR](2500);
 
BEGIN TRANSACTION 
 
  BEGIN TRY
 
    UPDATE [DBCATCH].[dbo].[TU_ActivityLog]                         WITH (ROWLOCK)
       SET [System]                  = ISNULL(@p_System,      [System]),
           [Error_ID]                = ISNULL(@p_Error_ID,    [Error_ID]),
           [DB]                      = ISNULL(@p_DB,          [DB]),
           [Obj]                     = ISNULL(@p_Obj,         [Obj]),
           [App]                     = ISNULL(@p_App,         [App]),
           [USER]                    = ISNULL(@p_User,        [USER]),
           [Description]             = ISNULL(@p_Description, [Description]),
           [StartTime]               = ISNULL(@p_StartTime,   [StartTime]),
           [EndTime]                 = ISNULL(@p_EndTime,     [EndTime]),
           [Active]                  = ISNULL(@p_Active,      [Active]),
           [DateModified]            = ISNULL(@p_DateModified, GETDATE())
     WHERE [ActivityLog_ID]          = @p_ActivityLog_ID;
 
  END TRY
 
  BEGIN CATCH
 
    SELECT @p_Obj                    = '[DBCATCH].[dbo].[TU_ActivityLog]';
 
    SELECT @v_Error                  = 100002
          ,@v_MSG                    = CONVERT(NVARCHAR(2500), 
                                       'Error: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_NUMBER(), -1)) + 
                                       ' Severity: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_SEVERITY(), -1)) +
                                       ' State: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_STATE(), -1)) +
                                       ' Line: ' +  CONVERT([NVARCHAR](255), ISNULL(ERROR_LINE(), -1)) +
                                       ' Procedure: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_PROCEDURE(), @p_Obj)) +
                                       ' MSG: Error Updating [DBCATCH].[dbo].[TU_ActivityLog] with error:' + ISNULL(ERROR_MESSAGE(), ''));  
 
    EXEC [DBCATCH].[dbo].[PU_ErrorLog_ADD]
      @p_Error_ID                    = @v_Error,
      @p_System                      = @p_System,
      @p_DB                          = @p_DB,
      @p_Obj                         = @p_Obj,
      @p_App                         = @p_App,
      @p_User                        = @p_User,
      @p_ErrorMsg                    = @v_MSG;
 
    ROLLBACK TRAN;
    RETURN @v_Error;
 
  END CATCH;
 
COMMIT TRANSACTION;
GO
 
IF OBJECT_ID('[dbo].[PU_ActivityLog_UPD]') IS NOT NULL
BEGIN
	PRINT '<<< CREATED PROCEDURE [dbo].[PU_ActivityLog_UPD] >>>';
END;
ELSE 
BEGIN
	PRINT '<<< FAILED TO CREATE PROCEDURE [dbo].[PU_ActivityLog_UPD] >>>';
END;
GO

Once again, we changed the error number and the custom message for this routine. Plan on always updating these two areas on the template as you create each new procedure. Speaking of the template, it’s time to add some code to the template so that we can add this feature, writing to the activity log.

USE DBCATCH
GO
 
IF EXISTS(SELECT 1
            FROM [dbo].[sysobjects] (NOLOCK)
           WHERE [id]          = OBJECT_ID(N'[dbo].[<Name, SYSNAME, PU_DB_ADD>]')
             AND OBJECTPROPERTY([id], N'IsProcedure')            = 1) 
 
BEGIN 
  DROP PROCEDURE [dbo].[<Name, SYSNAME, PU_DB_ADD>];
 
  PRINT '>>> DROPPED PROCEDURE [dbo].[<Name, SYSNAME, PU_DB_ADD>] <<<';
END 
GO
 
CREATE PROC [dbo].[<Name, SYSNAME, PU_DB_ADD>]
WITH ENCRYPTION
AS
 
/******************************************************/ 
/* Name        : <Name, SYSNAME, PU_DB_ADD>           */
/* Version     : 1.0                                  */
/* Author      :                                      */
/* Date        : 2019-07-24                           */
/* Description :                                      */
/******************************************************/ 
/* Date        : Version: Who: Description            */
/******************************************************/ 
/* 2019-07-24  : 1.0    :    : Initial Release.      */
/******************************************************/ 
 
SET NOCOUNT ON;
 
/* Declare Variables */
DECLARE @v_Error                     [INT]
       ,@v_MSG                       [NVARCHAR](500)
       ,@v_DB                        [NVARCHAR](100)
       ,@v_Obj                       [NVARCHAR](100)
       ,@v_App                       [NVARCHAR](100)
       ,@v_User                      [NVARCHAR](100)
       ,@v_Spid                      [NVARCHAR](100)
       ,@v_ActivityLog_ID            [INT]
       ,@v_ADesc                     [NVARCHAR](512)
       ,@v_Now                       [DATETIME]
       ,@v_LastErrorDate             [DATETIME]
       ,@v_SystemName                [NVARCHAR](100)
       ,@v_RowCount                  [INT]
       ,@v_Count                     [INT]
       ,@v_MinINT                    [INT]
       ,@v_MaxINT                    [INT]
       ,@v_Version                   [INT]       
       ,@v_KeyWord                   [SYSNAME]   
       ,@v_MSSQL_ID                  [INT]
       ,@v_Domain                    [NVARCHAR](100)  
       ,@v_Batch_Type_ID             [INT] 
	   ,@v_System                    [NVARCHAR](100)
       ,@v_SQL                       [NVARCHAR](4000)  
       ,@v_UserName                  [NVARCHAR](100)
       ,@v_PW                        [NVARCHAR](100)
       ,@v_Debug                     [INT];
 
SELECT @v_Now                        = GETDATE()
      ,@v_DB                         = DB_NAME()
      ,@v_Obj                        = '[DBCATCH].[dbo].[<Name, SYSNAME, PU_DB_ADD>]'
      ,@v_App                        = APP_NAME()
      ,@v_User                       = ISNULL(ORIGINAL_LOGIN(), USER_NAME()) 
      ,@v_Spid                       = CONVERT([NVARCHAR](25), @@SPID)
	  ,@v_System                     = HOST_NAME()
      ,@v_Debug                      = 0;  
 
 
/* Create Temp Tables */
 
IF @v_Debug                          = 1
BEGIN -- debug = 1 Ref: 1
 
  SELECT 'REF:1 - Define Static Variables';
 
  SELECT @v_Now                      AS [GETDATE]
        ,@v_DB                       AS [DB]
        ,@v_Obj                      AS [OBJ]
        ,@v_App                      AS [APP]
        ,@v_User                     AS [USER]
        ,@v_Spid                     AS [SPID];
 
END; -- debug = 1    
 
/* Start the activity_log */
EXEC [DBCATCH].[dbo].[PU_ActivityLog_ADD]
  @p_Error_ID                        = NULL,
  @p_System                          = @v_System,
  @p_DB                              = @v_DB,
  @p_Obj                             = @v_Obj,
  @p_App                             = @v_App,
  @p_User                            = @v_User,
  @p_Spid                            = @v_Spid,
  @p_Description                     = NULL,
  @p_StartTime                       = @v_Now,
  @p_EndTime                         = NULL,
  @p_ActivityLog_ID                  = @v_ActivityLog_ID OUTPUT;   
 
 
  BEGIN TRY
 
--==>> Start Code Here!
 
 
  END TRY
 
  BEGIN CATCH
 
    SELECT @v_Error                  = 100000
          ,@v_MSG                    = CONVERT(NVARCHAR(2500), 
                                       'Error: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_NUMBER(), -1)) + 
                                       ' Severity: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_SEVERITY(), -1)) +
                                       ' State: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_STATE(), -1)) +
                                       ' Line: ' +  CONVERT([NVARCHAR](255), ISNULL(ERROR_LINE(), -1)) +
                                       ' Procedure: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_PROCEDURE(), @v_Obj)) +
                                       ' MSG: Error from [DBCATCH].[dbo].[TU_Errorlog] with error:' + ISNULL(ERROR_MESSAGE(), ''));  
 
    EXEC [DBCATCH].[dbo].[PU_ErrorLog_ADD]
      @p_Error_ID                    = @v_Error,
      @p_System                      = @v_System,
      @p_DB                          = @v_DB,
      @p_Obj                         = @v_Obj,
      @p_App                         = @v_App,
      @p_User                        = @v_User,
      @p_ErrorMsg                    = @v_MSG;
 
    EXEC [DBCATCH].[dbo].[PU_ActivityLog_UPD]
      @p_ActivityLog_ID              = @v_ActivityLog_ID,
      @p_EndTime                     = @v_Now,
      @p_Error_ID                    = @v_Error,
      @p_Active                      = 0,
      @p_Description                 = @v_MSG,
      @p_DateModified                = @v_Now; 
 
	SELECT @v_Msg; 
    RETURN @v_Error;
 
  END CATCH;
 
EOF: 
 
/* Cleanup */
 
/* Update the activity Log */
EXEC [DBCATCH].[dbo].[PU_ActivityLog_UPD]
  @p_ActivityLog_ID                  = @v_ActivityLog_ID,
  @p_EndTime                         = @v_Now,
  @p_Active                          = 0,
  @p_DateModified                    = @v_Now;
 
GO   
 
IF OBJECT_ID('[dbo].[<Name, SYSNAME, PU_DB_ADD>]') IS NOT NULL
BEGIN
	PRINT '>>> CREATED PROCEDURE [dbo].[<Name, SYSNAME, PU_DB_ADD>] <<<';
END;
ELSE 
BEGIN
	PRINT '>>> FAILED TO CREATE PROCEDURE [dbo].[<Name, SYSNAME, PU_DB_ADD>] <<<';
END;

Let’s examine the changes in detail. First, we added the Insert procedure call right after checking our debug values and right before we start developing in our Begin TRY statement.

EXEC [DBCATCH].[dbo].[PU_ActivityLog_ADD]
  @p_Error_ID                        = NULL,
  @p_System                          = @v_System,
  @p_DB                              = @v_DB,
  @p_Obj                             = @v_Obj,
  @p_App                             = @v_App,
  @p_User                            = @v_User,
  @p_Spid                            = @v_Spid,
  @p_Description                     = NULL,
  @p_StartTime                       = @v_Now,
  @p_EndTime                         = NULL,
  @p_ActivityLog_ID                  = @v_ActivityLog_ID OUTPUT;

This starts the activity log with the values we defined and used for our error engine. We created an output param for the new identity and stuffed it into @v_ActivityLog_ID so that when we are at the end of the routine, we can update the row uniquely which is what the update proc expects as one of its input params.

Call to Update Procedure

The next change is placing a call to the update procedure right after the error log insert in the TRY CATCH segment as shown below:if you're seeing this, you're building a t-sql activity log!

 

 

 

 

 

 

 

 

The main thing to notice here is that we already have all the information about an error. If an error does happen, and we are already at a point to insert in the error log, and we have to update the activity log anyway, why not insert the error with the activity log?

If we look at the activity log and the error is null, that’s great! Otherwise, we know exactly where to go without going anywhere else, and we know the date and time of the process. If there is an error, we exit the routine directly from the TRY CATCH.

Then, if the process completes successfully, we need to update the activity log with end dates and leave the error id NULL. This is done directly under the cleanup comment as this cleans up our processes getting ready to exit cleanly.

 

 

 

 

 

 

And that is it for the development!

Building the T-SQL Activity Log Demo

All we need to do now is create a new demo from our updated template. We cannot use our old demo because it does not have the activity log feature set we just plugged into our template framework. Let’s save our template, copy it to a new query window, press CTRL + Shift + M to name the procedure and lets name it the same…PU_Demo1.

Our framework has the plumbing to look if it already exists and recreate itself, so let’s not add more demos. We’ll keep the demo procs somewhat manageable. At the ==>> Start Code Here! point, let’s place the code:

SELECT 'This is Demo1';

We will leave the error number as 10000 but will change the message:

Error FROM [DBCATCH].[dbo].[TU_Errorlog] WITH error:

To:

Error FROM [PU_Demo1] WITH error:

And that’s it! Let’s compile and execute the procedure PU_Demo1 and see what we have:

Congrats, you have a t-sql activity log

 

 

 

We still have one record in the error log; that did not change. But, now we have a single record in our activity log that shows the start and end times, the who, what, when, and where from. The Error ID is null, so it appears that the process ran successfully. It looks like we have a functional feature added, but we will not know for sure until we test the error update. To do this we will change the code of PU_Demo1 from:

SELECT 'This is Demo1';

To:

SELECT 'This is Demo1'
    RAISERROR ('Message', 19, 1) WITH LOG

Recompile the PU_Demo1 code and then run the procedure so we can analyze the results:

you've successfully built a t-sql activity log!

 

 

 

Final Word, and Looking Ahead

We built the T-SQL Activity Log. We have two records in the Error log, and we have two records in the Activity Log. The other thing to notice: not only is the Error_ID populated with the correct error code we used to error out in the TRY CATCH, we also have a description which is the error message that we would also find in the error log.

At this stage we have successfully completed adding the activity logging feature to our framework. Our product is already able to help us develop and troubleshoot closer to an enterprise level, while at the same time reducing the time we have to develop tedious tasks. Well done. Come back as we finish our feature set in T-SQL Framework part 4, where we will add Batching as a feature set to our framework.

GO TO PART FOUR

The post The T-SQL Activity Log: Framework, pt. 3 appeared first on SQL Solutions Group.

]]> https://sqlsolutionsgroup.com/the-tsql-activity-log/feed/ 0 The SQL Error Engine: Framework, pt. 2 https://sqlsolutionsgroup.com/sql-error-engine/ https://sqlsolutionsgroup.com/sql-error-engine/#respond Mon, 05 Aug 2019 11:32:06 +0000 http://sqlsolutionsgroup.com/?p=4165 The post The SQL Error Engine: Framework, pt. 2 appeared first on SQL Solutions Group.

]]> Building the SQL Error Engine

In this second post focused on a T-SQL framework (find part one here), we will build the plumbing to leverage our first feature of the framework, the error engine. This feature  will runs like an engine and do the manual work we do not want to do. To make it dynamic, the feature must be data-driven using small tools that work with each other. 

We will address architecture, database design, functions, and other coding principles, following some best practices. We are not going to describe each one in detail because that is out of scope for this project. That would make this post even longer. If you find that we have glanced over any topic that you feel should be discussed in more detail, please leave some feedback. If there is enough interest, we may publish a blog solely on that topic.

With this blog post, we will:

  1. Create a new database to hold our framework
  2. Create code tables, an error definition table, and an error logging table
  3. Populate our base data-driven tables (the code tables and the error definition table)
  4. Create a user-defined function to get our codes for us
  5. Add a new error message within the SQL engine
  6. Create a new template that will leverage the coding effort above

A lot of work is ahead of us, and there won’t necessarily be a lot of explanation.  For instance, we will create a database that will hold all stored procedures that will be leveraged to execute against your user databases. Why not just develop this in the user database for ease? Several reasons:

  1. We want to develop this once and only once. We come from an environment where one SQL developer may be tasked with writing for six .net developers and we don’t want to introduce confusion, so just one connection string. The developers make the connection to the code database, and the stored procs that are executed in that code database will be executed against the user data that returns result sets accordingly. In short, all roads lead to Rome.
  2. If all our security is set for an empty database with just a bunch of code, if someone were to gain access uninvited, what do they have? And they have even less if that code is encrypted.
  3. It forces the DBA to use a three-part naming scheme of database name, schema name, and object name. While this may look like it takes more time referencing each object in the database, we have noticed the precision results in a dramatic decrease in troubleshooting time. You get used to the extra typing (especially since intelligence is your friend), and when you are developing enterprise solutions you don’t want to be called out for being bush league when a few more seconds make you into an enterprise surgical developer (right?).

You can choose any name you wish to call the database. At SQL Solutions Group, we develop several internal products against the database we are calling DBCatch, so we will be using that. You can also use the name DBCatch and we encourage that. Just as long as you do not sell anything under that name without discussing that with us and the copyright thereof. 

Create Database and Tables

We are not going to get too fancy with the database options; keeping it simple with this code:

CREATE DATABASE DBCatch;
GO

 

Great, the first task is complete. Next, we need to create a few tables. The first set of tables will be the code tables. I do not like creating different code tables for every function that comes up, so I create two tables that act as a kind of named value pair. That way, these two tables take care of almost all my code definition needs. The first table is the Code Group (this is the header), and the second is the Code (this is the detail).  Let’s just jump in and you will see what I am talking about once we populate the tables and you can see them in action.

USE DBCatch;
GO
 
CREATE TABLE [dbo].[TU_CodeGroup]
(
  [CodeGroup_ID]    [INT]  IDENTITY (1, 1)   NOT NULL
    CONSTRAINT [PK_CodeGroup_ID]   
    PRIMARY KEY CLUSTERED
 ,[Name]            [NVARCHAR](100)          NOT NULL
 ,[DESC]            [NVARCHAR](255)          NOT NULL
    CONSTRAINT DF_TU_CodeGroup_Desc
    DEFAULT ('Group Code')
 ,[Active]          [BIT]                    NOT NULL
    CONSTRAINT DF_TU_CodeGroup_Active
    DEFAULT ((1))
 ,[DateCreated]     [DATETIME]               NOT NULL
    CONSTRAINT DF_TU_CodeGroup_DateCreated
    DEFAULT (GETDATE())
 ,[DateModified]    [DATETIME]               NOT NULL
    CONSTRAINT DF_TU_CodeGroup_DateModified
    DEFAULT (GETDATE())
) ON [PRIMARY];
GO
 
 
CREATE TABLE [dbo].[TU_Code]
(
  [Code_ID]         [INT]  IDENTITY (1, 1)   NOT NULL 
    CONSTRAINT [PK_Code_ID]   
    PRIMARY KEY CLUSTERED
 ,[CodeGroup_ID]    [INT]                    NOT NULL
    CONSTRAINT [FK_TU_Code_CodeGroup_ID]
       FOREIGN KEY ([CodeGroup_ID]) 
    REFERENCES [TU_CodeGroup]([CodeGroup_ID])
 ,[Name]            [NVARCHAR](100)          NOT NULL
 ,[DESC]            [NVARCHAR](255)          NOT NULL
    CONSTRAINT [DF_TU_Code_Desc]
    DEFAULT 'Code'
 ,[Active]          [BIT]                    NOT NULL
    CONSTRAINT [DF_TU_Code_Active]
    DEFAULT 1
 ,[DateCreated]     [DATETIME]               NOT NULL
    CONSTRAINT [DF_TU_Code_DateCreated]
    DEFAULT GETDATE()
 ,[DateModified]    [DATETIME]               NOT NULL
    CONSTRAINT [DF_TU_Code_DateModified]
    DEFAULT GETDATE()
) ON [PRIMARY];
GO

 

Naming Scheme and Populating Tables

At SSG, we adhere to a specific naming scheme. I recommend anyone that develops code should adopt a naming scheme that works, for you and your company. The prefix TU_ above in front of the table names stands for user table. SSG does a lot of data dictionary work and when we describe our objects, we like them to be grouped nicely. This way we can describe each set of objects (like tables, views, procedures, etc.) in sets visually. Now that our code tables are in the DBCatch database, populate them with the data we need:

USE DBCatch;
GO
 
  SET IDENTITY_INSERT TU_CodeGroup ON;
 
 
INSERT [DBCATCH].[dbo].[TU_CodeGroup]
(      CodeGroup_ID, Name,                    [DESC])
SELECT 1,            'ERROR',                 'ERROR CODES';
 
 
  SET IDENTITY_INSERT TU_CodeGroup OFF;
 
 
  SET IDENTITY_INSERT TU_Code ON;
 
INSERT [DBCATCH].[dbo].[TU_Code]
(      Code_ID, CodeGroup_ID, Name,                    [DESC])
SELECT 100,     1,            'DEFAULT',               'UNKNOWN ERROR'
UNION
SELECT 101,     1,            'SQL',                   'INTERNAL SQL SERVER RELATED ERROR'
UNION
SELECT 102,     1,            'DBCATCH',               'DBCATCH RELATED ERROR'
UNION 
SELECT 103,     1,            'ESCLATION',             'ERROR THAT HAS BEEN ESCALATED TO A PARENT ERROR'
UNION   
SELECT 104,     1,            'MAINTENANCE',           'DATABASE MAINTENANCE RELATED ERROR';
 
 
  SET IDENTITY_INSERT TU_Code OFF;

 

You can see the base data we just populated, with the two select statements below. This will make more sense once we create the user-defined function and actually start using it in the template.

USE DBCatch;
GO
 
SELECT [CodeGroup_ID]
      ,[Name]
	  ,[DESC]
	  ,[Active]
	  ,[DateCreated]
	  ,[DateModified]
  FROM [DBCatch].[dbo].[TU_CodeGroup];
 
SELECT [Code_ID]
      ,[CodeGroup_ID]
      ,[Name]
	  ,[DESC]
	  ,[Active]
	  ,[DateCreated]
	  ,[DateModified]
  FROM [DBCatch].[dbo].[TU_Code];

 

Trapping Errors with the SQL Error Engine

Now, let’s dive into the user-defined scalar function. This retrieves the code IDs for us when we use them programmatically within our solution. The idea here is that we will be trapping any errors that come up by a try catch method inside our T_SQL code. Once we capture the error, we categorize it and define it ourselves. This way, the error is not cryptic and we can define it in a way that will point us to the issue. This cuts down our troubleshooting time.

First, we’ll create the function. Next, we will continue making this solution more customized.

USE DBCATCH;
GO
 
SET NOCOUNT ON;
 
IF EXISTS(SELECT 1
            FROM [INFORMATION_SCHEMA].[ROUTINES] (NOLOCK)
           WHERE [SPECIFIC_SCHEMA]   = 'dbo'
             AND [SPECIFIC_NAME]     = 'FSU_Code_GET')
BEGIN
  DROP FUNCTION [dbo].[FSU_Code_GET];
 
  PRINT '<<< DROPPED FUNCTION [dbo].[FSU_Code_GET] >>>';
END;
GO
 
CREATE FUNCTION [dbo].[FSU_Code_GET] 
(	
  @p_CodeGroup                       [NVARCHAR](100)
 ,@p_Code                            [NVARCHAR](100)
)
RETURNS INT
WITH ENCRYPTION
AS
BEGIN
 
/************************************************************************/
/* Name	  : FSU_Code_GET                                          */
/* Version      : 1.0                                                   */
/* Author       : Jared Kirkpatrick                                     */
/* Date	  : 2019-07-24                                            */
/* Description  : Get the value from the code table.                    */
/************************************************************************/
/* Date         : Version: Who: Description                             */
/************************************************************************/
/* 2019-05-24   : 1.0    : JSK: Initial Release.                        */
/************************************************************************/
 
  DECLARE @v_Return                  [INT];
 
  SELECT @v_Return                   = b.[Code_ID]
    FROM [DBCATCH].[dbo].[TU_CodeGroup] a 
    JOIN [DBCATCH].[dbo].[TU_Code]      b 
      ON a.[CodeGroup_ID]            = b.[CodeGroup_ID]
   WHERE a.[Name]                    = @p_CodeGroup
     AND b.[Name]                    = @p_Code;
 
  RETURN @v_Return;
 
END;
GO
 
IF OBJECT_ID('[dbo].[FSU_Code_GET]') IS NOT NULL
BEGIN
	PRINT '<<< CREATED FUNCTION [dbo].[FSU_Code_GET] >>>';
END;
ELSE 
BEGIN
	PRINT '<<< FAILED TO CREATE FUNCTION [dbo].[FSU_Code_GET] >>>';
END;
GO

 

To see what we have done here, let’s look at the two tables we first populated, TU_CodeGroup and TU_Code.

sql error engine tables

The TU_CodeGroup table has one record, and the value (actually, the {Name} field), is “ERROR”. In the TU_Code table we will be mostly using the code_id that corelates to the “DBCatch” value since we are aiming at categorizing errors we trap from DBCatch. If we want the code for DBCatch errors, we use our function which takes the value of the code group (Error) and the value of the code (“DBCatch”) and we retrieve it by leveraging the function as shown below which will give us the value 102…

USE DBCatch;
GO
 
SELECT dbo.[FSU_Code_GET]('Error', 'DBCatch');

 

Next, we create the table that will hold the error definition, and then we will populate those definitions.

USE DBCatch;
GO
 
CREATE TABLE [dbo].[TU_Error]
(
  [Error_ID]     [INT]  IDENTITY (1, 1)   NOT NULL 
    CONSTRAINT [PK_Error_ID]   
    PRIMARY KEY CLUSTERED
 ,[Type_ID]      [INT]                    NOT NULL
    CONSTRAINT [FK_TU_Error_Code_ID]
       FOREIGN KEY ([Type_ID]) 
    REFERENCES [TU_Code]([Code_ID])
 ,[Name]         [NVARCHAR](100)          NOT NULL
 ,[Severity]     [INT]                    NULL
 ,[STATE]        [INT]                    NULL
 ,[DESC]         [NVARCHAR](255)          NOT NULL
    CONSTRAINT [DF_TU_Error_Desc]
    DEFAULT 'Error'
 ,[Active]       [BIT]                    NOT NULL
    CONSTRAINT [DF_TU_Error_Active]
    DEFAULT 1
 ,[DateCreated]  [DATETIME]               NOT NULL
    CONSTRAINT [DF_TU_Error_DateCreated]
    DEFAULT GETDATE()
 ,[DateModified] [DATETIME]               NOT NULL
    CONSTRAINT [DF_TU_Error_DateModified]
    DEFAULT GETDATE()
) ON [PRIMARY];
 
USE DBCatch;
GO
 
  SET IDENTITY_INSERT TU_Error ON;
 
INSERT [DBCATCH].[dbo].[TU_Error]
(      Error_ID, [Type_ID], [Name],          [Severity], [STATE], [DESC])
SELECT 100000,   100,       'DEFAULT_DBCATCH',   '16',      '1',     'UNKNOWN ERROR IN DBCATCH'
UNION  
SELECT 100001,   102,       'DBCATCH_INSERT',    '16',      '1',     'ERROR INSERTING INTO A DBCATCH TABLE'
UNION  
SELECT 100002,   102,       'DBCATCH_UPDATE',    '16',      '1',     'ERROR UPDATING A DBCATCH TABLE'
UNION  
SELECT 100003,   102,       'DBCATCH_DELETE',    '16',      '1',     'ERROR DELETING FROM A DBCATCH TABLE'
UNION  
SELECT 100004,   102,       'DBCATCH_SELECT',    '16',      '1',     'ERROR DBCATCH DATA COLLECTOR (SELECTING INFO)';
 
 
 
  SET IDENTITY_INSERT TU_Error OFF;
GO

 

The final step for the definition is to add an error message in the SQL engine, but we are going to put a twist on this so it is not exactly straightforward. We set the message to be defined at runtime so that we can customize the error message and manipulate. This helps us understand what is happening.

When we get to the template that shows how this works, it will be clear. It is hard to describe what you cannot see, but we will have a demo at the end so that you can see it in action. Then the light bulbs will go off in that “a-ha!” moment.

Let’s make some error messages!

USE DBCatch;
GO
 
IF EXISTS(SELECT 1
            FROM [master].[sys].[messages] 
           WHERE [message_id] = 100000)
BEGIN
  EXEC [DBCATCH].[dbo].[SP_DROPMESSAGE] @msgnum = 100000;
END;  
 
EXEC [DBCATCH].[dbo].[SP_ADDMESSAGE] 
  @msgnum = 100000,
  @severity = 16,
  @msgtext = N'%s';
 
IF EXISTS(SELECT 1
            FROM [master].[sys].[messages] 
           WHERE [message_id] = 100001)
BEGIN
  EXEC [DBCATCH].[dbo].[SP_DROPMESSAGE] @msgnum = 100001;
END;  
 
EXEC [DBCATCH].[dbo].[SP_ADDMESSAGE] 
  @msgnum = 100001,
  @severity = 16,
  @msgtext = N'%s';
 
IF EXISTS(SELECT 1
            FROM [master].[sys].[messages] 
           WHERE [message_id] = 100002)
BEGIN
  EXEC [DBCATCH].[dbo].[SP_DROPMESSAGE] @msgnum = 100002;
END;  
 
EXEC [DBCATCH].[dbo].[SP_ADDMESSAGE] 
  @msgnum = 100002,
  @severity = 16,
  @msgtext = N'%s'; 
 
IF EXISTS(SELECT 1
            FROM [master].[sys].[messages] 
           WHERE [message_id] = 100003)
BEGIN
  EXEC [DBCATCH].[dbo].[SP_DROPMESSAGE] @msgnum = 100003;
END;  
 
EXEC [DBCATCH].[dbo].[SP_ADDMESSAGE] 
  @msgnum = 100003,
  @severity = 16,
  @msgtext = N'%s'; 
 
IF EXISTS(SELECT 1
            FROM [master].[sys].[messages] 
           WHERE [message_id] = 100004)
BEGIN
  EXEC [DBCATCH].[dbo].[SP_DROPMESSAGE] @msgnum = 100004;
END;  
 
EXEC [DBCATCH].[dbo].[SP_ADDMESSAGE] 
  @msgnum = 100004,
  @severity = 16,
  @msgtext = N'%s'; 
 
GO

 

One Last Thing….

Before we create our template for the SQL error engine, we create one more table. This holds our error information that will be logged. We also create a simple stored proc to insert into that table. Any time we have an error we will log that error into this table. If you have the stored proc in management studio, that is great to see the error printed for you. But, in most cases a process will be running when you are not looking. Having this log will be helpful because we can look at it after the fact and see what happened. The code for the error log table is:

USE DBCatch;
GO
 
CREATE TABLE [dbo].[TU_ErrorLog]
(
  [ErrorLog_ID]     [INT]  IDENTITY (1, 1)     NOT NULL 
    CONSTRAINT [PK_ErrorLog_ID]   
    PRIMARY KEY CLUSTERED
 ,[Error_ID]        [INT]                      NOT NULL
    CONSTRAINT [FK_TU_ErrorLog_Error_ID]
       FOREIGN KEY ([Error_ID]) 
    REFERENCES [TU_Error]([Error_ID])   
 ,[System]			[NVARCHAR](100)            NULL 
 ,[DB]              [NVARCHAR](100)            NULL
 ,[Obj]             [NVARCHAR](100)            NULL
 ,[App]             [NVARCHAR](100)            NULL
 ,[USER]            [NVARCHAR](100)            NULL
 ,[SPID]            [NVARCHAR](100)            NULL
 ,[ErrorMsg]        [NVARCHAR](2500)           NULL 
 ,[Active]          [BIT]                      NOT NULL
    CONSTRAINT [DF_TU_ErrorLog_Active]
    DEFAULT 1
 ,[DateCreated]     [DATETIME]                 NOT NULL
    CONSTRAINT [DF_TU_ErrorLog_DateCreated]
    DEFAULT GETDATE()
 ,[DateModified]    [DATETIME]                 NOT NULL
    CONSTRAINT [DF_TU_ErrorLog_DateModified]
    DEFAULT GETDATE()
) ON [PRIMARY];
 
GO
 
USE DBCATCH
GO
 
IF EXISTS(SELECT 1
            FROM [dbo].[sysobjects] (NOLOCK)
           WHERE [id]                                            = OBJECT_ID(N'[dbo].[PU_ErrorLog_ADD]')
             AND OBJECTPROPERTY([id], N'IsProcedure')            = 1) 
 
BEGIN 
  DROP PROCEDURE [dbo].[PU_ErrorLog_ADD]; 
 
  PRINT '<<< DROPPED PROCEDURE [dbo].[PU_ErrorLog_ADD] >>>';
END 
GO
 
CREATE PROC [dbo].[PU_ErrorLog_ADD]
  @p_Error_ID                        [INT]                       = NULL,
  @p_System                          [NVARCHAR](100)             = NULL,
  @p_DB                              [NVARCHAR](100)             = NULL,
  @p_Obj                             [NVARCHAR](100)             = NULL,
  @p_App                             [NVARCHAR](100)             = NULL,
  @p_User                            [NVARCHAR](100)             = NULL,
  @p_SPID                            [NVARCHAR](100)             = NULL,
  @p_ErrorMsg                        [NVARCHAR](2500)            = NULL
WITH ENCRYPTION
AS
 
/*************************************************************************************************/ 
/* Name        : PU_ErrorLog_ADD                                                                 */
/* Version     : 1.0                                                                             */
/* Author      : Jared Kirkpatrick                                                               */
/* Date        : 2019-07-24                                                                      */
/* Description : Inserts an error into the errorlog table                                        */
/*************************************************************************************************/ 
/* Date        : Version: Who: Description                                                       */
/*************************************************************************************************/ 
/* 2019-07-24  : 1.0    : REB: Initial Release.                                                  */
/*************************************************************************************************/
 
SET NOCOUNT ON;
 
/* Declare Variables */
DECLARE @v_Error                     [INT]
       ,@v_MSG                       [NVARCHAR](2500)
 
IF @p_Error_ID                       IS NULL
BEGIN
  SELECT @p_Error_ID                 = 0;
END  
 
BEGIN TRANSACTION 
 
  INSERT [DBCATCH].[dbo].[TU_ErrorLog]
  (
    [Error_ID],
    [System],
    [DB],
    [Obj],
    [App],
    [USER],
    [ErrorMsg],
    [SPID]
  )
  VALUES
  (
    @p_Error_ID,
    @p_System,
    @p_DB,
    @p_Obj,
    @p_App,
    @p_User,
    @p_ErrorMsg,
    @p_SPID 
  )
 
  SELECT @v_Error                    = @@ERROR;
 
  IF @v_Error <> 0
  BEGIN
 
    RAISERROR('INSERT into TU_ErrorLog Failed!', 16, 1)
    ROLLBACK TRANSACTION
    RETURN @v_Error;
 
  END;
 
COMMIT TRANSACTION;
GO
 
IF OBJECT_ID('[dbo].[PU_ErrorLog_ADD]') IS NOT NULL
BEGIN
	PRINT '<<< CREATED PROCEDURE [dbo].[PU_ErrorLog_ADD] >>>';
END;
ELSE 
BEGIN
	PRINT '<<< FAILED TO CREATE PROCEDURE [dbo].[PU_ErrorLog_ADD] >>>';
END;
GO

 

If you have been following along, your DBCatch database should have the following objects when you look at it in Management Studio:

sql error engine

Producing the Template

We’ve put it all together. Now we can create (or better yet, save off) a template that has the feature we just created in a stored procedure. Before we get into the template, we will create additional features, so this template will change as we develop.

USE DBCATCH
GO
 
IF EXISTS(SELECT 1
            FROM [dbo].[sysobjects] (NOLOCK)
           WHERE [id]          = OBJECT_ID(N'[dbo].[<Name, SYSNAME, PU_DB_ADD>]')
             AND OBJECTPROPERTY([id], N'IsProcedure')            = 1) 
 
BEGIN 
  DROP PROCEDURE [dbo].[<Name, SYSNAME, PU_DB_ADD>];
 
  PRINT '>>> DROPPED PROCEDURE [dbo].[<Name, SYSNAME, PU_DB_ADD>] <<<';
END 
GO
 
CREATE PROC [dbo].[<Name, SYSNAME, PU_DB_ADD>]
WITH ENCRYPTION
AS
 
/******************************************************/ 
/* Name        : <Name, SYSNAME, PU_DB_ADD>           */
/* Version     : 1.0                                  */
/* Author      :                                      */
/* Date        : 2019-07-24                           */
/* Description :                                      */
/******************************************************/ 
/* Date        : Version: Who: Description            */
/******************************************************/ 
/* 2019-07-24  : 1.0    :    : Initial Release.       */
/******************************************************/ 
 
SET NOCOUNT ON;
 
/* Declare Variables */
DECLARE @v_Error                     [INT]
       ,@v_MSG                       [NVARCHAR](500)
       ,@v_DB                        [NVARCHAR](100)
       ,@v_Obj                       [NVARCHAR](100)
       ,@v_App                       [NVARCHAR](100)
       ,@v_User                      [NVARCHAR](100)
       ,@v_Spid                      [NVARCHAR](100)
       ,@v_ActivityLog_ID            [INT]
       ,@v_ADesc                     [NVARCHAR](512)
       ,@v_Now                       [DATETIME]
       ,@v_LastErrorDate             [DATETIME]
       ,@v_SystemName                [NVARCHAR](100)
       ,@v_RowCount                  [INT]
       ,@v_Count                     [INT]
       ,@v_MinINT                    [INT]
       ,@v_MaxINT                    [INT]
       ,@v_Version                   [INT]       
       ,@v_KeyWord                   [SYSNAME]   
       ,@v_MSSQL_ID                  [INT]
       ,@v_Domain                    [NVARCHAR](100)  
       ,@v_Batch_Type_ID             [INT] 
	   ,@v_System                    [NVARCHAR](100)
       ,@v_SQL                       [NVARCHAR](4000)  
       ,@v_UserName                  [NVARCHAR](100)
       ,@v_PW                        [NVARCHAR](100)
       ,@v_Debug                     [INT];
 
SELECT @v_Now                        = GETDATE()
      ,@v_DB                         = DB_NAME()
      ,@v_Obj                        = '[DBCATCH].[dbo].[<Name, SYSNAME, PU_DB_ADD>]'
      ,@v_App                        = APP_NAME()
      ,@v_User                       = ISNULL(ORIGINAL_LOGIN(), USER_NAME()) 
      ,@v_Spid                       = CONVERT([NVARCHAR](25), @@SPID)
	  ,@v_System                     = HOST_NAME()
      ,@v_Debug                      = 0;  
 
 
/* Create Temp Tables */
 
IF @v_Debug                          = 1
BEGIN -- debug = 1 Ref: 1
 
  SELECT 'REF:1 - Define Static Variables';
 
  SELECT @v_Now                      AS [GETDATE]
        ,@v_DB                       AS [DB]
        ,@v_Obj                      AS [OBJ]
        ,@v_App                      AS [APP]
        ,@v_User                     AS [USER]
        ,@v_Spid                     AS [SPID];
 
END; -- debug = 1    
 
 
  BEGIN TRY
 
--==>> Start Code Here!
 
 
  END TRY
 
  BEGIN CATCH
 
    SELECT @v_Error                  = 100000
          ,@v_MSG                    = CONVERT(NVARCHAR(2500), 
                                       'Error: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_NUMBER(), -1)) + 
                                       ' Severity: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_SEVERITY(), -1)) +
                                       ' State: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_STATE(), -1)) +
                                       ' Line: ' +  CONVERT([NVARCHAR](255), ISNULL(ERROR_LINE(), -1)) +
                                       ' Procedure: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_PROCEDURE(), @v_Obj)) +
                                       ' MSG: Error from [DBCATCH].[dbo].[TU_Errorlog] with error:' + ISNULL(ERROR_MESSAGE(), ''));  
 
    EXEC [DBCATCH].[dbo].[PU_ErrorLog_ADD]
      @p_Error_ID                    = @v_Error,
      @p_System                      = @v_System,
      @p_DB                          = @v_DB,
      @p_Obj                         = @v_Obj,
      @p_App                         = @v_App,
      @p_User                        = @v_User,
      @p_ErrorMsg                    = @v_MSG;
 
	SELECT @v_Msg; 
    RETURN @v_Error;
 
  END CATCH;
 
EOF: 
 
/* Cleanup */
 
GO   
 
IF OBJECT_ID('[dbo].[<Name, SYSNAME, PU_DB_ADD>]') IS NOT NULL
BEGIN
	PRINT '>>> CREATED PROCEDURE [dbo].[<Name, SYSNAME, PU_DB_ADD>] <<<';
END;
ELSE 
BEGIN
	PRINT '>>> FAILED TO CREATE PROCEDURE [dbo].[<Name, SYSNAME, PU_DB_ADD>] <<<';
END;
GO

 

Yep, it’s a lot when you first look at it, but we are going to go through it and it will be manageable. If you work with it every day, it will become second nature. The first thing we need to do is change the template parameters. This is embedded in the template and looks like this:

This is the code from the very top, and instead of the actual procedure name we see: 

[<Name, SYSNAME, PU_DB_ADD>]

To change this parameter, we simultaneously hold down the keys SHIFT/CTRL/M and then type in the name of the stored procedure we are creating. For instance, if we want to create a demo, we could type PU_Demo1 (the prefix PU stands for user defined procedure).

 

After you have changed the parameters, go down about 35 lines to the variables…

SELECT @v_Now                        = GETDATE()
      ,@v_DB                         = DB_NAME()
      ,@v_Obj                        = '[DBCATCH].[dbo].[PU_Demo1]'
      ,@v_App                        = APP_NAME()
      ,@v_User                       = ISNULL(ORIGINAL_LOGIN(), USER_NAME()) 
      ,@v_Spid                       = CONVERT([NVARCHAR](25), @@SPID)
      ,@v_System                     = HOST_NAME()
      ,@v_Debug                      = 0;

 

We defined the variable @v_obj when we set the parameters above. The system functions define the rest of the variables. If you look closely, they may look familiar. Remember the table TU_Errorlog from above? Here’s the table definition again:

CREATE TABLE [dbo].[TU_ErrorLog]
(
  [ErrorLog_ID]     [INT]  IDENTITY (1, 1)     NOT NULL 
    CONSTRAINT [PK_ErrorLog_ID]   
    PRIMARY KEY CLUSTERED
 ,[Error_ID]        [INT]                      NOT NULL
    CONSTRAINT [FK_TU_ErrorLog_Error_ID]
       FOREIGN KEY ([Error_ID]) 
    REFERENCES [TU_Error]([Error_ID])   
 ,[System]			[NVARCHAR](100)            NULL 
 ,[DB]              [NVARCHAR](100)            NULL
 ,[Obj]             [NVARCHAR](100)            NULL
 ,[App]             [NVARCHAR](100)            NULL
 ,[USER]            [NVARCHAR](100)            NULL
 ,[SPID]            [NVARCHAR](100)            NULL
 ,[ErrorMsg]        [NVARCHAR](2500)           NULL 
 ,[Active]          [BIT]                      NOT NULL
    CONSTRAINT [DF_TU_ErrorLog_Active]
    DEFAULT 1
 ,[DateCreated]     [DATETIME]                 NOT NULL
    CONSTRAINT [DF_TU_ErrorLog_DateCreated]
    DEFAULT GETDATE()
 ,[DateModified]    [DATETIME]                 NOT NULL
    CONSTRAINT [DF_TU_ErrorLog_DateModified]
    DEFAULT GETDATE()
) ON [PRIMARY];
 
GO

 

Do some of the variables look familiar? We are not at that “a-ha!” moment yet, but I am hoping you are connecting the dots. These variables are being defined dynamically at run time that will be used to populate the error log table when we call the TU_ErrorLog_ADD Procedure. 

The last part of the variables, defined above, which needs your attention is the @v_Debug value that is set to 0. When developing your stored procedure you may want to set this to 1 and use this technique throughout your logic. Copy and paste the if statement and supply your own run time variables you will be using as shown below:

IF @v_Debug                          = 1
BEGIN -- debug = 1 Ref: 1
 
  SELECT 'REF:1 - Define Static Variables';
 
  SELECT @v_Now                      AS [GETDATE]
        ,@v_DB                       AS [DB]
        ,@v_Obj                      AS [OBJ]
        ,@v_App                      AS [APP]
        ,@v_User                     AS [USER]
        ,@v_Spid                     AS [SPID];
 
END; -- debug = 1

 

Kicking the Tires

Now that the variables are set, we go to the Begin Try / End Try part of the template where we see a comment:  Start Code Here! I bet you cannot guess where you start your logic code of the stored procedure… Yeah, it really is that simple. When you are done with your stored procedure and it is error-free, then you are done. That’s it.

BEGIN TRY
 
--==>> Start Code Here!
 
 
  END TRY

 

If by chance you do get an error, then we go into the error trapping and error message customization part of the template called the begin catch and the end catch. (We love the try and catch functionality in stored procedures. Where do you think the name DBCatch comes from?)

BEGIN CATCH
 
    SELECT @v_Error   = 100000
          ,@v_MSG     = CONVERT(NVARCHAR(2500), 
  'Error: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_NUMBER(), -1)) + 
  ' Severity: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_SEVERITY(), -1)) +
  ' State: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_STATE(), -1)) +
  ' Line: ' +  CONVERT([NVARCHAR](255), ISNULL(ERROR_LINE(), -1)) +
  ' Procedure: ' + CONVERT([NVARCHAR](255), ISNULL(ERROR_PROCEDURE(), @v_Obj)) +
  ' MSG: Error from [DBCATCH].[dbo].[TU_Errorlog] with error:' + ISNULL(ERROR_MESSAGE(), ''));  
 
    EXEC [DBCATCH].[dbo].[PU_ErrorLog_ADD]
      @p_Error_ID                    = @v_Error,
      @p_System                      = @v_System,
      @p_DB                          = @v_DB,
      @p_Obj                         = @v_Obj,
      @p_App                         = @v_App,
      @p_User                        = @v_User,
      @p_ErrorMsg                    = @v_MSG;
 
	SELECT @v_Msg; 
    RETURN @v_Error;
 
  END CATCH;

 

Once we capture the error, we take our defined variables from above and concatenate a run time message that should give us some insight about the object in question. Using some system error functions we create a meaningful message that should help us determine what’s going on.

It then takes the information and calls the error log stored proc, which  adds the record to the table. Then it prints the message just in case you are calling it from management studio. It may look like a lot, but once you break it down into its individual pieces it is really pretty easy to consume. Let’s finish this segment by creating a simple demo that will error out and we can see our framework in action.

Demo1

First, we take our template and hold the keys SHIFT/CTRL/M and name the stored procedure PU_Demo1.

Second we go to the line where it says start code here:

–==>> Start Code Here!

And we replace it with:

SELECT 'This is Demo1'
    RAISERROR ('Message', 19, 1) WITH LOG

 

Third, we compile the stored procedure by clicking on the execute button, and then we are run the procedure by opening a new query window and typing:

USE DBCatch;
GO
 
EXEC PU_Demo1;
 
GO

 

Once we execute that we will see the following message below:

 

Lastly, if we go to the error log table we should see a single record like that below:

Conclusion

We really did not get some earth-shattering enlightenment about the error with our SQL error engine. But lets face it, all we did is raiserror with nothing going on so there was not much to go on.

But, we do have the information about where the error happened (TU_Demo1), from where (management studio) and from what system. You could say this Demo was the worst-case solution in terms of getting a great message out of the error (I wanted to keep the demo as simple as we could and show the worst case). This at least gives us a pretty good starting point. Especially if your solution had 100 stored procs wrapped in its process running in a job every night. I hope you are thinking where you can start using this functionality as we come to a close of the second part of this series.

Come back when we extend the functionality of this framework by adding a running activity log that will interact with our already running error handler.

GO TO PART THREE

The post The SQL Error Engine: Framework, pt. 2 appeared first on SQL Solutions Group.

]]> https://sqlsolutionsgroup.com/sql-error-engine/feed/ 0 Building a T-SQL Framework https://sqlsolutionsgroup.com/t-sql-framework/ https://sqlsolutionsgroup.com/t-sql-framework/#respond Wed, 02 Mar 2016 14:22:07 +0000 http://sqlsolutionsgroup.com/?p=2869 Depending on the objective, building a framework allows the architect to imbed the fundamental segments of a structure so that the existing requirements can be built on top of the foundation, leveraging the functionality of the framework. This usually includes core functionality that repeats consistently through a process like Logging, Error handling, Successes/Failure Notifications, Import/Export […]

The post Building a T-SQL Framework appeared first on SQL Solutions Group.

]]> Depending on the objective, building a framework allows the architect to imbed the fundamental segments of a structure so that the existing requirements can be built on top of the foundation, leveraging the functionality of the framework. This usually includes core functionality that repeats consistently through a process like Logging, Error handling, Successes/Failure Notifications, Import/Export Validations, and File System  processes  just to name a few.

There are several different types of frameworks. ETL Frameworks have been receiving the most popularity lately, but they can extend to T-SQL, Analysis Services, Reporting Services, CLR, and even PowerShell within Microsoft SQL Server Data Services. Usually all frameworks will have a simple set of core processes that would benefit any new process by embedding the procedure in question to run consistently through a mechanism such as a template that would allow the extension of logic code from developers.

Today we are going to start a multi-step project to create a basic T-SQL framework from the ground up. This framework will have three components, including:

  • Logging – Each process that runs in the framework will log its activity to a table. This feature can also be leveraged later if you created an ETL framework or extended to add additional information according to your business needs.
  • Batching – Each T-sql procedure will be grouped into a batch. This is another extension of logging but it is also the the same foundation in which to build batching row sets for insert, update, delete, or even selects.
  • Error Handling – The framework will incorporate some custom error handling that will identify, define, and log errors that happen through the framework.

In the end we will create a stored procedure template that will facilitate the consistent execution of the frameworks features while allowing the developer to code the logic necessary for their task at hand.

In this section of the project we are going to look at creating a new database, the table structures necessary to support the framework, and populate the database with base data (insert into the code tables), that will allow our framework to be data driven.

The first thing we are going to do is create a database called DBCatch.  The defaults will be fine for the database with the statement below:

USE master;
GO
 
 
IF NOT EXISTS(SELECT 1
                FROM master.sys.databases
	        WHERE name = 'DBCatch')
BEGIN
 
  CREATE DATABASE DBCatch;
 
END;
GO

Now let’s take a quick look at the tables that we will create and give an explanation of each one. This is a great place to start a data dictionary but since that is outside of our scope we will not be defining that today. Here is a look at the database diagram once we have all our tables built:

T-sql framework diagram

First, start with the code tables CodeGroup and Code. We do not want to clutter our database with all the code tables that will be needed to make the database data driven so we combine them into a name value pair. Let’s take a phone code example for instance. If we were storing phone numbers, then the overall group name would be phone so we would make an entry into the CodeGroup called phone. Next, we take the primary key that was created for the group phone and we carry it over to the Code table when we define what kind of phone entry we are inserting like:

  • Cell Phone
  • Home Phone
  • Work Phone
  • Fax
  • Toll Free Phone
  • Alternate Phone

This type of code works for 90% of the codes and definitions we want, but we need another structure for everything else. We will use a technique called the registry table. Back in the day, some developers used the windows registry to store look up data but that became problematic. Great idea but the implementation left something to be desired, so a new idea of creating a structure/table like the registry was created and adopted by a few developers and by us today. The registry table is made up of a RegKey (Registry Key),  like for example if you wanted an email list we could have a RegKey of:

HKEY_LOCAL_MACHINE\SOFTWARE\DBCatch\MAINT\DBMAIL\RECIPIENTS

And then the RegValue (Registry Value) of the emails that make up the email list. Now, one last thing that is important about this table is that it is not encrypted because everyone has their own favorite flavor of encryption, and the nature of this type of data is usually somewhat sensitive (Data paths, email addresses, Powershell directories, etc.). We create a simple level of obscurity by converting the character values into varbinary and then back to character when we need to use them. This is not a bad thing because we will design the framework to do so seamlessly.

The first feature set of tables are the error handling tables. We would normally start with our number one feature we defined (logging), but it is dependent on the error handling functionality in order to be complete. The error tables are Error which will define the error, and ErrorLog which will hold a record for each error that is thrown. In the Error table we will first have to go to our code tables and create a group (CodeGroup), called “Error”, and then define the codes (Code), of the error that will be caught such as:

  • Default or Unknown
  • SQL – Internal sql server error
  • DBCATCH generated error
  • INSERT
  • UPDATE
  • DELETE
  • T-SQL (usually for dynamic sql)
  • ESCALATION – error that has been escalated to a parent error
  • MAINTENACE – a maintenance related error

The primary key for the Code table will be set in the Foreign Key column “Type_ID” that describes the type of error we are defining. The additional attributes Name, Severity, State, and Desc will be described in detail in the next segment when we define the logic code for the Error and ErrorLog tables.

The next table is the Activity log or ActivityLog.  It will hold a record for every procedure that is executed within the framework, and if there is an error with the procedure it will show that here as well. The activity log will log the who, what, wherefrom, when, and how long did it take. This information will be obtained at runtime through the template.

The last set of records are the “Batch” and “BatchDetail” tables that will extend the logging into groups so that we can see where in a process things go wrong, slow down, or just what groups of statements are being executed together. Although we are only dealing with stored procedure batches at the moment, we are taking into account that we will want to extend the functionality to batch records, or even processes at a future date so we will populate a type_id foreign key that reflects the code for stored procedure batch. The rest of the attributes will be defined in later segments when the logic code is created and the template is designed to wrap the functionality up into a working framework.

Now that we have a brief explanation of the tables we are going to provide the DDL to create the tables in the T-SQL Statement below.

/************************************************************************/
/* Name	        : Create DBCatch Table Structures                       */
/* Version      : 1.0                                                   */
/* Author       : Jared Kirkpatrick                                     */
/* Date	        : 2015-11-15                                            */
/* Description	: Create Table Structures                               */
/************************************************************************/
/* Date         : Version: Who: Description                             */
/************************************************************************/
/* 2015-11-15   : 1.0    : JSK: Initial Release.                        */
/************************************************************************/
 
 
/************************************************************************/
/* ::NOTES::														    */
/************************************************************************/
 
 
USE DBCatch;
GO
 
DECLARE @v_Mode INT
 
SELECT @v_Mode = 1 -- 0 = drop only 1 = drop/create 2 = create only
 
IF @v_Mode < 2
BEGIN
 
 
/*****************************BASE FRAMEWORK*********************************/
	IF OBJECT_ID(N'[dbo].[ErrorLog]', 'u') IS NOT NULL 
	BEGIN
  		PRINT CHAR(10) + CHAR(13) + 'Dropping table [dbo].[ErrorLog].';	
		DROP TABLE [dbo].[ErrorLog];
	END;
 
/*****************************BASE FRAMEWORK*********************************/
	IF OBJECT_ID(N'[dbo].[ActivityLog]', 'u') IS NOT NULL 
	BEGIN
  		PRINT CHAR(10) + CHAR(13) + 'Dropping table [dbo].[ActivityLog].';	
		DROP TABLE [dbo].[ActivityLog];
	END;
 
/*****************************BASE FRAMEWORK*********************************/
	IF OBJECT_ID(N'[dbo].[BatchDetail]', 'u') IS NOT NULL 
	BEGIN
  		PRINT CHAR(10) + CHAR(13) + 'Dropping table [dbo].[BatchDetail].';	
		DROP TABLE [dbo].[BatchDetail];
	END;
 
/*****************************BASE FRAMEWORK*********************************/
	IF OBJECT_ID(N'[dbo].[Batch]', 'u') IS NOT NULL 
	BEGIN
  		PRINT CHAR(10) + CHAR(13) + 'Dropping table [dbo].[Batch].';	
		DROP TABLE [dbo].[Batch];
	END;
 
/*****************************BASE FRAMEWORK*********************************/
	IF OBJECT_ID(N'[dbo].[Error]', 'u') IS NOT NULL 
	BEGIN
  		PRINT CHAR(10) + CHAR(13) + 'Dropping table [dbo].[Error].';	
		DROP TABLE [dbo].[Error];
	END;
 
/*****************************BASE FRAMEWORK*********************************/	
	IF OBJECT_ID(N'[dbo].[Registry]', 'u') IS NOT NULL
	BEGIN
  		PRINT CHAR(10) + CHAR(13) + 'Dropping table [dbo].[Registry].';	
		DROP TABLE [dbo].[Registry];
	END;	
 
/*****************************BASE FRAMEWORK*********************************/
	IF OBJECT_ID(N'[dbo].[Code]', 'u') IS NOT NULL 
	BEGIN
  		PRINT CHAR(10) + CHAR(13) + 'Dropping table [dbo].[Code].';	
		DROP TABLE [dbo].[Code];
	END;
 
/*****************************BASE FRAMEWORK*********************************/
	IF OBJECT_ID(N'[dbo].[CodeGroup]', 'u') IS NOT NULL 
	BEGIN
  		PRINT CHAR(10) + CHAR(13) + 'Dropping table [dbo].[CodeGroup].';	
		DROP TABLE [dbo].[CodeGroup];
	END;
 
END;	
 
 
IF @v_Mode >= 1
BEGIN   -- @v_Mode >= 1
 
/************************************************************************/
/* ::NOTES:  Create the objects                                        	*/
/************************************************************************/
 
/*================================================================================================*/
/****** Object:  Table [dbo].[CodeGroup]          Script Date: 2015-11-15                 ******/
/*================================================================================================*/
 
	PRINT CHAR(10) + CHAR(13) + 'Creating table [dbo].[CodeGroup].';	
 
CREATE TABLE [dbo].[CodeGroup]
(
  [CodeGroup_ID]               [INT]  IDENTITY (1, 1)                                        NOT NULL
    CONSTRAINT [PK_CodeGroup_ID]   
    PRIMARY KEY CLUSTERED
 ,[Name]                       [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NOT NULL
 ,[DESC]                       [NVARCHAR](255) COLLATE SQL_Latin1_General_CP1_CI_AS          NOT NULL
    CONSTRAINT DF_CodeGroup_Desc
    DEFAULT ('Group Code')
 ,[Active]                     [BIT]                                                         NOT NULL
    CONSTRAINT DF_CodeGroup_Active
    DEFAULT ((1))
 ,[DateCreated]                [DATETIME]                                                    NOT NULL
    CONSTRAINT DF_CodeGroup_DateCreated
    DEFAULT (GETDATE())
 ,[DateModified]               [DATETIME]                                                    NOT NULL
    CONSTRAINT DF_CodeGroup_DateModified
    DEFAULT (GETDATE())
) ON [PRIMARY];
 
/* Alt key Unique index */
CREATE UNIQUE NONCLUSTERED INDEX ak_CodeGroup_Name ON [dbo].[CodeGroup]([Name])
 
 
/*================================================================================================*/
/****** Object:  Table [dbo].[Code]            Script Date: 2015-11-15                    ******/
/*================================================================================================*/
 
	PRINT CHAR(10) + CHAR(13) + 'Creating table [dbo].[Code].';	
 
 
CREATE TABLE [dbo].[Code]
(
  [Code_ID]                    [INT]  IDENTITY (1, 1)                                        NOT NULL 
    CONSTRAINT [PK_Code_ID]   
    PRIMARY KEY CLUSTERED
 ,[CodeGroup_ID]               [INT]                                                         NOT NULL
    CONSTRAINT [FK_Code_CodeGroup_ID]
       FOREIGN KEY ([CodeGroup_ID]) 
    REFERENCES [dbo].[CodeGroup]([CodeGroup_ID])
 ,[Name]                       [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NOT NULL
 ,[DESC]                       [NVARCHAR](255) COLLATE SQL_Latin1_General_CP1_CI_AS          NOT NULL
    CONSTRAINT [DF_Code_Desc]
    DEFAULT 'Code'
 ,[Active]                     [BIT]                                                         NOT NULL
    CONSTRAINT [DF_Code_Active]
    DEFAULT 1
 ,[DateCreated]                [DATETIME]                                                    NOT NULL
    CONSTRAINT [DF_Code_DateCreated]
    DEFAULT GETDATE()
 ,[DateModified]               [DATETIME]                                                    NOT NULL
    CONSTRAINT [DF_Code_DateModified]
    DEFAULT GETDATE()
) ON [PRIMARY];
 
 
/* FK Index */
CREATE NONCLUSTERED INDEX idxfk_Code_CodeGroup_ID ON [dbo].[Code]([CodeGroup_ID])
 
 
/*================================================================================================*/
/****** Object:  Table [dbo].[Registry]        Script Date: 2015-11-15                    ******/
/*================================================================================================*/
IF OBJECT_ID(N'[dbo].[Registry]', 'u') IS NULL
BEGIN
 
	PRINT CHAR(10) + CHAR(13) + 'Creating table [dbo].[Registry].';	
 
CREATE TABLE [dbo].[Registry]
(
  [Registry_ID]			       [INT]  IDENTITY (1,1)                                         NOT NULL 
    CONSTRAINT [PK_Registry_ID]   
    PRIMARY KEY CLUSTERED
 ,[RegKey]				       [NVARCHAR](255) COLLATE SQL_Latin1_General_CP1_CI_AS	         NOT NULL
 ,[RegValue]				   [VARBINARY](255)                                              NOT NULL
 ,[Active]                     [BIT]                                                         NOT NULL
   CONSTRAINT [DF_Registry_Active]
   DEFAULT (1)
 ,[DateCreated]			       [DATETIME]      					                             NOT NULL
   CONSTRAINT [DF_Registry_DateCreated]
   DEFAULT GETDATE()
 ,[DateModified]			   [DATETIME]												     NOT NULL
   CONSTRAINT [DF_Registry_DateModified]
   DEFAULT GETDATE()
) ON [PRIMARY];
 
END;
 
 
/* Alt key Unique index */
CREATE UNIQUE NONCLUSTERED INDEX ak_Registry_RegKey ON [dbo].[Registry]([RegKey]) INCLUDE ([RegValue])
 
/*================================================================================================*/
/****** Object:  Table [dbo].[Error]           Script Date: 2015-11-15                    ******/
/*================================================================================================*/
 
	PRINT CHAR(10) + CHAR(13) + 'Creating table [dbo].[Error].';	
 
 
CREATE TABLE [dbo].[Error]
(
  [Error_ID]                   [INT]  IDENTITY (1, 1)                                        NOT NULL 
    CONSTRAINT [PK_Error_ID]   
    PRIMARY KEY CLUSTERED
 ,[Type_ID]                    [INT]                                                         NOT NULL
    CONSTRAINT [FK_Error_Code_ID]
       FOREIGN KEY ([Type_ID]) 
    REFERENCES [dbo].[Code]([Code_ID])
 ,[Name]                       [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NOT NULL
 ,[Severity]                   [INT]                                                         NULL
 ,[STATE]                      [INT]                                                         NULL
 ,[DESC]                       [NVARCHAR](255) COLLATE SQL_Latin1_General_CP1_CI_AS          NOT NULL
    CONSTRAINT [DF_Error_Desc]
    DEFAULT 'Error'
 ,[Active]                     [BIT]                                                         NOT NULL
    CONSTRAINT [DF_Error_Active]
    DEFAULT 1
 ,[DateCreated]                [DATETIME]                                                    NOT NULL
    CONSTRAINT [DF_Error_DateCreated]
    DEFAULT GETDATE()
 ,[DateModified]               [DATETIME]                                                    NOT NULL
    CONSTRAINT [DF_Error_DateModified]
    DEFAULT GETDATE()
) ON [PRIMARY];
 
 
/* FK Index */
CREATE NONCLUSTERED INDEX idxfk_Error_Type_ID ON [dbo].[Error]([Type_ID])
 
 
/*================================================================================================*/
/****** Object:  Table [dbo].[Batch]           Script Date: 2015-11-15                    ******/
/*================================================================================================*/
 
	PRINT CHAR(10) + CHAR(13) + 'Creating table [dbo].[Batch].';	
 
 
CREATE TABLE [dbo].[Batch]
(
  [Batch_ID]			       [UNIQUEIDENTIFIER]                                            NOT NULL
    CONSTRAINT [PK_Batch_ID]   
    PRIMARY KEY CLUSTERED
    CONSTRAINT [DF_Batch_ID]
    DEFAULT CONVERT(UNIQUEIDENTIFIER, CONVERT(BINARY(10), NEWID()) + 
                                      CONVERT(BINARY(6), GETDATE())) 
 ,[Type_ID]                    [INT]                                                         NOT NULL
    CONSTRAINT [FK_Batch_Type_ID]
       FOREIGN KEY ([Type_ID]) 
    REFERENCES [dbo].[Code]([Code_ID]) 
    CONSTRAINT [DF_Batch_Type_ID]
    DEFAULT 0    
 ,[Name]                       [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NOT NULL
    CONSTRAINT [DF_Batch_Name]
    DEFAULT 'Batch:' + CONVERT(NVARCHAR(10), GETDATE(), 110)    
 ,[Parent_ID]                  [INT]                                                         NULL
 ,[Active]                     [BIT]                                                         NOT NULL
    CONSTRAINT [DF_Batch_Active]
    DEFAULT 1
 ,[DateCreated]                [DATETIME]                                                    NOT NULL
    CONSTRAINT [DF_Batch_DateCreated]
    DEFAULT GETDATE()
 ,[DateModified]               [DATETIME]                                                    NOT NULL
    CONSTRAINT [DF_Batch_DateModified]
    DEFAULT GETDATE()
) ON [PRIMARY];
 
 
/* FK Index */
CREATE NONCLUSTERED INDEX idxfk_Batch_Type_ID ON [dbo].[Batch]([Type_ID])
 
 
/*================================================================================================*/
/****** Object:  Table [dbo].[BatchDetail]        Script Date: 2015-11-15                 ******/
/*================================================================================================*/
 
	PRINT CHAR(10) + CHAR(13) + 'Creating table [dbo].[BatchDetail].';	
 
 
CREATE TABLE [dbo].[BatchDetail]
(
  [BatchDetail_ID]			   [BIGINT]  IDENTITY (1, 1)                                     NOT NULL 
    CONSTRAINT [PK_BatchDetail_ID]   
    PRIMARY KEY CLUSTERED
 ,[Batch_ID]                   [UNIQUEIDENTIFIER]                                            NOT NULL
    CONSTRAINT [FK_BatchDetail_Batch_ID]
       FOREIGN KEY ([Batch_ID]) 
    REFERENCES [dbo].[Batch]([Batch_ID]) 
 ,[BatchSize]                  [INT]                                                         NOT NULL
    CONSTRAINT [DF_Batch_BatchSize]
    DEFAULT 0     
 ,[OBJECT_ID]                  [INT]                                                         NULL   
 ,[Entity_ID]                  [UNIQUEIDENTIFIER]                                            NULL   
 ,[OBJECT]                     [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[StartDate]                  [DATETIME]                                                    NULL
 ,[EndDate]                    [DATETIME]                                                    NULL
 ,[Rank]                       [INT]                                                         NOT NULL
    CONSTRAINT [DF_BatchDetail_Rank]
    DEFAULT 1
 ,[Active]                     [BIT]                                                         NOT NULL
    CONSTRAINT [DF_BatchDetail_Active]
    DEFAULT 1
 ,[DateCreated]                [DATETIME]                                                    NOT NULL
    CONSTRAINT [DF_BatchDetail_DateCreated]
    DEFAULT GETDATE()
 ,[DateModified]               [DATETIME]                                                    NOT NULL
    CONSTRAINT [DF_BatchDetail_DateModified]
    DEFAULT GETDATE()
) ON [PRIMARY];
 
 
/* FK Index */
CREATE NONCLUSTERED INDEX idxfk_BatchDetail_Batch_ID ON [dbo].[BatchDetail]([Batch_ID])
 
/*================================================================================================*/
/****** Object:  Table [dbo].[ActivityLog]         Script Date: 2015-11-15                ******/
/*================================================================================================*/
	PRINT CHAR(10) + CHAR(13) + 'Creating table [dbo].[ActivityLog].';	
 
CREATE TABLE [dbo].[ActivityLog] 
(
  [ActivityLog_ID]             [BIGINT]  IDENTITY (1, 1)                                     NOT NULL
    CONSTRAINT [PK_ActivityLog_ID]   
    PRIMARY KEY CLUSTERED  
 ,[Error_ID]                   [INT]                                                         NULL
    CONSTRAINT [FK_ActivityLog_Error_ID]
       FOREIGN KEY ([Error_ID]) 
    REFERENCES [dbo].[Error]([Error_ID])   
 ,[SystemName]                 [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[DB]                         [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[Obj]                        [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[App]                        [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[USER]                       [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL 
 ,[SPID]                       [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL 
 ,[Description]                [NVARCHAR](512)	COLLATE SQL_Latin1_General_CP1_CI_AS         NULL 
 ,[StartTime]                  [DATETIME]                                                    NULL
 ,[EndTime]                    [DATETIME]                                                    NULL 
 ,[Active]                     [BIT]                                                         NOT NULL
    CONSTRAINT [DF_ActivityLog_Active]
    DEFAULT (1)
 ,[DateCreated]			       [DATETIME]												     NOT NULL
    CONSTRAINT [DF_ActivityLog_DateCreated]
    DEFAULT GETDATE()
 ,[DateModified]			   [DATETIME]												     NOT NULL
    CONSTRAINT [DF_ActivityLog_DateModified]
    DEFAULT GETDATE()			
) ON [PRIMARY];
 
 
/* FK Index */
CREATE NONCLUSTERED INDEX idxfk_ActivityLog_Error_ID ON [dbo].[ActivityLog]([Error_ID])
 
/*================================================================================================*/
/****** Object:  Table [dbo].[ErrorLog]           Script Date: 2015-11-15                 ******/
/*================================================================================================*/
 
	PRINT CHAR(10) + CHAR(13) + 'Creating table [dbo].[ErrorLog].';	
 
 
CREATE TABLE [dbo].[ErrorLog]
(
  [ErrorLog_ID]                [INT]  IDENTITY (1, 1)                                        NOT NULL 
    CONSTRAINT [PK_ErrorLog_ID]   
    PRIMARY KEY CLUSTERED
 ,[Error_ID]                   [INT]                                                         NOT NULL
    CONSTRAINT [FK_ErrorLog_Error_ID]
       FOREIGN KEY ([Error_ID]) 
    REFERENCES [dbo].[Error]([Error_ID])   
 ,[DB]                         [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[Obj]                        [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[App]                        [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[USER]                       [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[SPID]                       [NVARCHAR](100) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[ErrorMsg]                   [NVARCHAR](2500)COLLATE SQL_Latin1_General_CP1_CI_AS          NULL 
 ,[Active]                     [BIT]                                                         NOT NULL
    CONSTRAINT [DF_ErrorLog_Active]
    DEFAULT 1
 ,[DateCreated]                [DATETIME]                                                    NOT NULL
    CONSTRAINT [DF_ErrorLog_DateCreated]
    DEFAULT GETDATE()
 ,[DateModified]               [DATETIME]                                                    NOT NULL
    CONSTRAINT [DF_ErrorLog_DateModified]
    DEFAULT GETDATE()
) ON [PRIMARY];
 
 
/* FK Index */
CREATE NONCLUSTERED INDEX idxfk_ErrorLog_Error_ID ON [dbo].[ErrorLog]([Error_ID])
 
/*================================================================================================*/
/****** Object:  Table [dbo].[Application]        Script Date: 2015-11-15                 ******/
/*================================================================================================*/
 
	PRINT CHAR(10) + CHAR(13) + 'Creating table [dbo].[Application].';	
 
 
CREATE TABLE [dbo].[Application]
(
  [Application_ID]             [INT]  IDENTITY (1, 1)                                        NOT NULL 
    CONSTRAINT [PK_Application_ID]   
    PRIMARY KEY CLUSTERED
 ,[ApplicationName]            [NVARCHAR](255) COLLATE SQL_Latin1_General_CP1_CI_AS          NOT NULL
    CONSTRAINT U_Applications_ApplicationName
	UNIQUE
 ,[AppDesc]                    [NVARCHAR](512) COLLATE SQL_Latin1_General_CP1_CI_AS          NULL
 ,[Active]                     [BIT]                                                         NOT NULL
    CONSTRAINT [DF_Application_Active]
    DEFAULT 1
 ,[DateCreated]                [DATETIME]                                                    NOT NULL
    CONSTRAINT [DF_Application_DateCreated]
    DEFAULT GETDATE()
 ,[DateModified]               [DATETIME]                                                    NOT NULL
    CONSTRAINT [DF_Application_DateModified]
    DEFAULT GETDATE()
) ON [PRIMARY];
 
 
 
 
END
GO

The last part of this segment is to populate the data-driven tables which are CodeGroup, Code, Registry, and Error. Now we are populating Error so that we can capture errors and put them in one of the groups we are defining. The other thing we are doing is creating new errors in SQL server but we are not completely defining them, what I mean is that the error message that is usually associate with an error is left out by design so that the message can be created at runtime for a more user friendly surgical message. This will be explained in much more detail in the last segment but take a look at the code because it will be referenced in the near future. The code is listed below but one last thing is that you must have elevated privileges (sa sysadmin) to run this script since it does add system level functionality:

USE [DBCatch];
GO
 
SET NOCOUNT ON;
 
DECLARE @v_Debug               [BIT]                          = 1 -- 1 = print all info msgs, 0 silent install
       ,@v_Error               [INT]                          = 0
       ,@v_NewLine             [NCHAR](2)                     = CHAR(13) + CHAR(10)
       ,@v_DynSQL2             [NCHAR](1)                     = CHAR(39)
       ,@v_MSG                 [NVARCHAR](2000)
       ,@v_SQL                 [NVARCHAR](4000);
 
 
/* Registry */
  IF @v_Debug = 1 
  BEGIN -- debug code
 
    SELECT 'Populating Table: [DBCatch].[dbo].[Registry]';
 
  END; -- debug code
 
/* Registry */
INSERT [DBCatch].[dbo].[Registry]
(RegKey, RegValue)
SELECT 'HKEY_LOCAL_MACHINE\SOFTWARE\DBCatch\MAINTVER', CONVERT(VARBINARY(255), '0.01')
UNION
SELECT 'HKEY_LOCAL_MACHINE\SOFTWARE\DBCatch\MAINT\DBMAIL\RECIPIENTS', CONVERT(VARBINARY(255), 'ja*************@***************up.com;')
UNION
SELECT 'HKEY_LOCAL_MACHINE\SOFTWARE\DBCatch\PSS\PATH', CONVERT(VARBINARY(255), 'C:\MSSQL\Code\DBCatch\PSScripts\')
UNION
SELECT 'HKEY_LOCAL_MACHINE\SOFTWARE\DBCatch\SSIS\PATH', CONVERT(VARBINARY(255), 'C:\MSSQL\Code\DBCatch\Projects\SSISConfig\')
UNION
SELECT 'HKEY_LOCAL_MACHINE\SOFTWARE\DBCatch\BATCH\ROWSET', CONVERT(VARBINARY(255), '25000')
UNION
SELECT 'HKEY_LOCAL_MACHINE\SOFTWARE\CLETL\EXTRACTHIST\SEC', CONVERT(VARBINARY(255), '300')
 
 
 
  IF @v_Debug = 1 
  BEGIN -- debug code
 
    SELECT '[DBCatch].[dbo].[Registry] VALUES';
 
    SELECT CONVERT([VARCHAR](2000), [RegValue])
          ,* 
      FROM [DBCatch].[dbo].[Registry] (NOLOCK);
 
	 -- select dbo.FSU_RegValue_GET ('HKEY_LOCAL_MACHINE\SOFTWARE\DBCatch\EXTRACTHIST\SEC')
 
  END; -- debug code 
 
/* Codes */
  IF @v_Debug = 1 
  BEGIN -- debug code
 
    SELECT 'Populating TABLE: [DBCatch].[dbo].[CodeGroup]';
 
  END; -- debug code
 
  SET IDENTITY_INSERT [dbo].[CodeGroup] ON;
 
INSERT [DBCatch].[dbo].[CodeGroup]
(      CodeGroup_ID, Name,                    [Desc])
SELECT 0,            'DEFAULT',               'TEMP/UNKNOWN CODE GROUP'
UNION
SELECT 1,            'ERROR',                 'ERROR CODES'
UNION
SELECT 2,            'BATCH',                 'TYPE OF BATCHS'
UNION
SELECT 4,            'LINEAGE',               'TYPE OF LINEAGE';
 
  SET IDENTITY_INSERT [dbo].[CodeGroup] OFF;
 
 
  IF @v_Debug = 1 
  BEGIN -- debug code
 
    SELECT '[DBCatch].[dbo].[CodeGroup] VALUES';
 
    SELECT * 
      FROM [DBCatch].[dbo].[CodeGroup] (NOLOCK);
 
  END; -- debug code
 
  IF @v_Debug = 1 
  BEGIN -- debug code
 
    SELECT 'Populating TABLE: [DBCatch].[dbo].[Code]';
 
  END; -- debug code
 
  SET IDENTITY_INSERT [dbo].[Code] ON;
 
INSERT [DBCatch].[dbo].[Code]
(      Code_ID, CodeGroup_ID, Name,                    [Desc])
SELECT 0,       0,            'DEFAULT',               'TEMP/UNKNOWN CODE'
UNION
SELECT 100,     1,            'DEFAULT',               'UNKNOWN ERROR'
UNION
SELECT 101,     1,            'SQL',                   'INTERNAL SQL SERVER RELATED ERROR'
UNION
SELECT 102,     1,            'DBCATCH',               'DBCatch RELATED ERROR'
UNION
SELECT 103,     1,            'INSERT',                'SQL TABLE INSERT RELATED ERROR'
UNION
SELECT 104,     1,            'UPDATE',                'SQL TABLE UPDATE RELATED ERROR'
UNION
SELECT 105,     1,            'DELETE',                'SQL TABLE DELETE RELATED ERROR'
UNION
SELECT 106,     1,            'T-SQL',                 'T-SQL RELATED ERROR'
UNION 
SELECT 107,     1,            'ESCLATION',             'ERROR THAT HAS BEEN ESCALATED TO A PARENT ERROR'
UNION   
SELECT 108,     1,            'MAINTENANCE',           'DATABASE MAINTENANCE RELATED ERROR'
UNION
SELECT 200,     2,            'DEFAULT',               'DEFAULT BATCH TYPE'
UNION
SELECT 201,     2,            'PROCEDURE',             'SP BATCH FOR ORDINAL COLLECTION'
UNION
SELECT 202,     2,            'RECORDCOUNT',           'SET NUMBER OF RECORDS TO BE BATCHED'
UNION
SELECT 400,     4,            'DEFAULT',               'DEFAULT LINEAGE'
UNION
SELECT 401,     4,            'TABLE',                 'SOURCE TABLE LINEAGE';
 
 
  SET IDENTITY_INSERT [dbo].[Code] OFF;
 
  IF @v_Debug = 1 
  BEGIN -- debug code
 
    SELECT '[DBCatch].[dbo].[Code] VALUES';
 
    SELECT * 
      FROM [DBCatch].[dbo].[Code] (NOLOCK);
 
  END; -- debug code
 
  IF @v_Debug = 1 
  BEGIN -- debug code
 
    SELECT 'Populating TABLE: [DBCatch].[dbo].[Error]';
 
  END; -- debug code
 
  SET IDENTITY_INSERT [dbo].[Error] ON;
 
INSERT [DBCatch].[dbo].[Error]
(      Error_ID, [Type_ID], [Name],              [Severity],[State], [Desc])
SELECT 0,        100,       'DEFAULT',           '16',      '1',     'UNKNOWN/TEMP ERROR'
UNION
SELECT 100000,   102,       'DEFAULT_DBCatch',   '16',      '1',     'UNKNOWN/TEMP ERROR IN DBCatch'
UNION  
SELECT 100001,   102,       'DBCatch_INSERT',    '16',      '1',     'ERROR INSERTING INTO A TABLE'
UNION  
SELECT 100002,   102,       'DBCatch_UPDATE',    '16',      '1',     'ERROR UPDATING A TABLE'
UNION  
SELECT 100003,   102,       'DBCatch_DELETE',    '16',      '1',     'ERROR DELETING FROM A TABLE'
UNION  
SELECT 100004,   102,       'DBCatch_SELECT',    '16',      '1',     'ERROR DBCatch DATA COLLECTOR (SELECTING INFO)'
UNION  
SELECT 100005,   102,       'DBCatch_ASSEMBLY',  '16',      '1',     'ERROR DBCatch C# SP/FN'
UNION
SELECT 100006,   102,       'DBCatch_WMIC',      '16',      '1',     'ERROR DBCatch WMIC Query'
UNION  
SELECT 100007,   102,       'ERRORENGINE',       '16',       '1',    'ERROR WITHIN THE FOUNDATION'
UNION  
SELECT 100008,   102,       'BACKUP',            '16',       '1',     'ERROR WITH A BACKUP'
UNION  
SELECT 100009,   102,       'SQLJOB',            '16',       '1',     'ERROR WITH A SQL JOB'
UNION  
SELECT 100010,   104,       'REINDEX',           '16',       '1',     'ERROR EXECUTING DATABASE REINDEXING MAINTENANCE UTILITY';
 
  SET IDENTITY_INSERT [dbo].[Error] OFF;
 
IF @v_Debug = 1 
BEGIN -- debug code
 
  SELECT '[DBCatch].[dbo].[Error] VALUES';
 
  SELECT * 
    FROM [DBCatch].[dbo].[Error] (NOLOCK);
 
END; -- debug code
 
/* Add error Messages */
IF EXISTS(SELECT 1
            FROM [master].[sys].[messages] 
           WHERE [message_id] = 100001)
BEGIN
  EXEC [DBCatch].[dbo].[sp_dropmessage] @msgnum = 100001;
END;  
 
EXEC [DBCatch].[dbo].[sp_addmessage] 
  @msgnum = 100001,
  @severity = 16,
  @msgtext = N'%s';

We are well on our way to having a basic T-SQL framework that is scalable enough when completed that it can be extended itself or incorporated into a new framework outside of just stored procedures.

The post Building a T-SQL Framework appeared first on SQL Solutions Group.

]]> https://sqlsolutionsgroup.com/t-sql-framework/feed/ 0 Weaning yourself off of SQL Profiler (Part 1) https://sqlsolutionsgroup.com/wean-off-sql-profiler-part-1/ https://sqlsolutionsgroup.com/wean-off-sql-profiler-part-1/#comments Tue, 09 Jun 2015 06:26:24 +0000 http://sqlsolutionsgroup.com/?p=2336 In this brave, new world of Extended Events (XE, XEvents), I find myself with a mixture of scripts for troubleshooting issues – some use XE, and some use traces. We’ve all been told that XE is a much better system (it is much more lightweight, causing less of an issue with the server). In fact, […]

The post Weaning yourself off of SQL Profiler (Part 1) appeared first on SQL Solutions Group.

]]> NoProfilerIn this brave, new world of Extended Events (XE, XEvents), I find myself with a mixture of scripts for troubleshooting issues – some use XE, and some use traces. We’ve all been told that XE is a much better system (it is much more lightweight, causing less of an issue with the server). In fact, it is so much better that Microsoft has deprecated SQL Trace and SQL Profiler, and in the future, one will not be able to run any traces at all. Believe it or not, this is a good thing!

TSQL2sDay150x150It just so happens that today is the 67th installment of the monthly T-SQL Tuesday blogging event. T-SQL Tuesday, that wonderful monthly blogging party started by Adam Machanic where a selected host challenges the SQL Server universe to have a blog post about a specific topic. This wild frenzy of SQL Server blog posting occurs on the second Tuesday of each month. This month, it is being hosted by my friend (and the person with the highest amount of energy known to mankind) Jes Borland (b / t), and the topic that she has chosen is Extended Events. Her specific challenge to the SQL Server universe is:

I want to know (and others do, too) how you’ve solved problems with Extended Events. What sessions have you created? What unique way have you used predicates or targets? What challenges have you overcome?

The biggest challenge that I have is not a technical challenge… it’s a personal challenge: actually getting started with XE. I have so many scripts for doing traces, that I just immediately use them instead of the better XE system. I really need to wean myself off of using Profiler / traces. Therefore, I’ve decided to start converting my trace scripts into XE scripts, and I’ll share with you how I go about doing it. Today, I’m going to look at my favorite trace script – a trace to capture deadlock information.

First off, let’s start with the trace script:

-- Create a trace to capture deadlocks
DECLARE @rc INTEGER,
        @TraceID INTEGER,
        @maxfilesize BIGINT,
        @OutputFileName NVARCHAR(256),
        @TraceStopTime DATETIME,
        @ON BIT,
        @intfilter INT,
        @bigintfilter BIGINT;
 
SET @ON = 1;
SET @maxfilesize = 10; --mb 
DECLARE @FileName NVARCHAR(256);
-- added InstanceName for when server is running multiple instances
SET @FileName = N'C:\SQL\Traces\' + CONVERT(sysname, SERVERPROPERTY('InstanceName')) + '\' +
               CONVERT(CHAR(8), GETDATE(), 112) + '_' +
               REPLACE(CONVERT(CHAR(8), GETDATE(), 114), ':', '');
EXECUTE xp_create_subdir @FileName;               
SET @FileName = @FileName + N'\' + REPLACE(@@SERVERNAME, N'\', N'_') + '_DeadlockTrace_' + CONVERT(CHAR(8), GETDATE(), 112) + N'_' + REPLACE(CONVERT(CHAR(8), GETDATE(), 114), N':', N'') + N'Z';
PRINT @FileName;
SET @TraceStopTime = DATEADD(HOUR, 6, GETDATE());
EXEC @rc = sp_trace_create @TraceID output, 2, @FileName, @maxfilesize, @TraceStopTime;
 
IF (@rc != 0) GOTO error
 
-- Set the events
-- Event 148: Deadlock Graph
EXEC sp_trace_setevent @TraceID, 148, 1, @on;   -- TextData
EXEC sp_trace_setevent @TraceID, 148, 11, @on;  -- LoginName
EXEC sp_trace_setevent @TraceID, 148, 12, @on;  -- SPID
EXEC sp_trace_setevent @TraceID, 148, 14, @on;  -- StartTime
-- Event 25: Lock:Deadlock
EXEC sp_trace_setevent @TraceID, 25, 1, @on;    -- TextData
EXEC sp_trace_setevent @TraceID, 25, 2, @on;    -- BinaryData
EXEC sp_trace_setevent @TraceID, 25, 6, @on;    -- NTUserName
EXEC sp_trace_setevent @TraceID, 25, 9, @on;    -- ClientProcessID
EXEC sp_trace_setevent @TraceID, 25, 10, @on;   -- ApplicationName
EXEC sp_trace_setevent @TraceID, 25, 11, @on;   -- LoginName
EXEC sp_trace_setevent @TraceID, 25, 12, @on;   -- SPID
EXEC sp_trace_setevent @TraceID, 25, 13, @on;   -- Duration
EXEC sp_trace_setevent @TraceID, 25, 14, @on;   -- StartTime
EXEC sp_trace_setevent @TraceID, 25, 15, @on;   -- EndTime
EXEC sp_trace_setevent @TraceID, 25, 22, @on;   -- ObjectID
EXEC sp_trace_setevent @TraceID, 25, 32, @on;   -- Mode
EXEC sp_trace_setevent @TraceID, 25, 35, @on;   -- DatabaseName
EXEC sp_trace_setevent @TraceID, 25, 57, @on;   -- Type
-- Event 59: Lock:Deadlock Chain
EXEC sp_trace_setevent @TraceID, 59, 1, @on;    -- TextData
EXEC sp_trace_setevent @TraceID, 59, 2, @on;    -- BinaryData
EXEC sp_trace_setevent @TraceID, 59, 12, @on;   -- SPID
EXEC sp_trace_setevent @TraceID, 59, 14, @on;   -- StartTime
EXEC sp_trace_setevent @TraceID, 59, 22, @on;   -- ObjectID
EXEC sp_trace_setevent @TraceID, 59, 32, @on;   -- Mode
EXEC sp_trace_setevent @TraceID, 59, 35, @on;   -- DatabaseName
EXEC sp_trace_setevent @TraceID, 59, 57, @on;   -- Type
 
 
-- Set the Filters
-- Set the trace status to start
EXEC sp_trace_setstatus @TraceID, 1;
 
-- display trace id for future references
SELECT TraceID=@TraceID;
GOTO finish;
 
error: 
SELECT ErrorCode=@rc;
 
finish: 
GO

From this script, you can see that the trace is collecting data from three events (Deadlock Graph (148), Lock:Deadlock (25) and Lock:Deadlock Chain (59)) and several columns for each event. The next step is to convert all of this information into XE events / actions. For this, I’ll modify the script in BOL (at https://msdn.microsoft.com/en-us/library/ff878264.aspx) to the following to return the XE events / actions for a specified running trace. Since the trace needs to be running, execute the above trace script to start the trace, then run the following code:

-- change the following value to the trace_id for the running trace to be converted to XE
DECLARE @trace_id INTEGER = 2;
 
SELECT DISTINCT
        tb.trace_event_id,
        te.name AS 'Event Class',
        em.package_name AS 'Package',
        em.xe_event_name AS 'XEvent Name',
        tb.trace_column_id,
        tc.name AS 'SQL Trace Column',
        am.xe_action_name AS 'Extended Events action'
FROM    sys.FN_TRACE_GETEVENTINFO(@trace_id) ei
        JOIN sys.trace_events te
            ON te.trace_event_id = ei.eventid
        JOIN sys.trace_columns tc  -- all available trace columns
            ON tc.trace_column_id = ei.columnid
        LEFT OUTER JOIN sys.trace_xe_event_map em
            ON em.trace_event_id = te.trace_event_id
        LEFT OUTER JOIN sys.trace_event_bindings tb
            ON tb.trace_event_id = em.trace_event_id
            AND tc.trace_column_id = tb.trace_column_id
        LEFT OUTER JOIN sys.trace_xe_action_map am
            ON am.trace_column_id = tc.trace_column_id
ORDER BY tb.trace_event_id, tb.trace_column_id;

These results may contain NULL values in the “Extended Events action” column. A NULL value here means that there is not a corresponding event action for that column. For my deadlock trace, I get the following results:

SSMS2014XE00-TraceXExref

With this information, I can now jump into SSMS (2014) and use the New Session wizard to create a new XE session (Expand the Management and Extended Events nodes. Right click on Sessions and select New Session). I name the session “Deadlocks”, and click the “Events” page to select the events that I want to use.

SSMS2014XE01-NewXESession

Since all of the events that I’m looking for contain “Deadlock”, I search for this and 5 events are displayed. I double-click each of the three listed above (or select them and click the “>” button) to move them to the “Selected Events” grid.

SSMS2014XE02-SelectEvents

The next step is to configure the columns, so click the Configure button.

SSMS2014XE03-ConfigureEvents

Select the lock_deadlock event, and on the “Global Fields (Actions)” tab select the client_app_name, client_pid, database_name, nt_username, server_principal_name and session_id columns.

SSMS2014XE04-ConfigureEventColumns

Click on the “Event Fields” tab, and check the checkboxes for database_name and resource_description. Fields that show up in this tab with checkboxes have customizable actions, and don’t actually collect the information unless checked. Frequently this will be because these fields require an additional level of resource usage to gather this information.

SSMS2014XE05-ConfigureEventColumns

Select the lock_deadlock_chain event, and select the session_id and database_name columns on the “Global Fields (Actions)” tab. Click on the “Event Fields” tab and check the checkbox for database_name and resource_description. Finally, select the xml_deadlock_report event, and select the server_principal_name and session_id columns on the “Global Fields (Actions)” tab.

Finally, click on the Data Storage page, and add an event_file target type. The file name defaults to the name that you gave the session earlier. I set the maximum size to 250mb and 5 rollover files. If you select to script to a new query window (HIGHLY recommended), you will have a script with all of the DDL necessary to create this XE.

SSMS2014XE06-ConfigureDataStorage

At this point, there are two changes that I like to make. First, I like to make my scripts so that they won’t have an error, and if the XE session already exists then one will be generated. To make it bullet-proof, I add an IF EXISTS check at the top and drop the XE session if it already exists. Secondly, I like to call xp_create_subdir to create the directory that the script points to, just in case the directory doesn’t exist. Note that xp_create_subdir will be successful if the directory already exists, so I don’t check to ensure that the directory doesn’t exist before executing this procedure.

My final XE Deadlock script looks like:

IF EXISTS (SELECT 1 FROM sys.server_event_sessions WHERE name = 'Deadlocks')
	DROP EVENT SESSION [Deadlocks] ON SERVER;
GO
EXECUTE xp_create_subdir 'C:\SQL\XE_Out';
GO
CREATE EVENT SESSION [Deadlocks]
ON SERVER
ADD EVENT sqlserver.lock_deadlock(
    SET collect_database_name=(1),collect_resource_description=(1)
	ACTION 
	(
			  sqlserver.client_app_name	-- ApplicationName from SQLTrace
			, sqlserver.client_pid	-- ClientProcessID from SQLTrace
			, sqlserver.nt_username	-- NTUserName from SQLTrace
			, sqlserver.server_principal_name	-- LoginName from SQLTrace
			, sqlserver.session_id	-- SPID from SQLTrace
	)
),
ADD EVENT sqlserver.lock_deadlock_chain(
    SET collect_database_name=(1),collect_resource_description=(1)
	ACTION 
	(
			  sqlserver.session_id	-- SPID from SQLTrace
	)
),
ADD EVENT sqlserver.xml_deadlock_report(
	ACTION 
	(
			  sqlserver.server_principal_name	-- LoginName from SQLTrace
			, sqlserver.session_id	-- SPID from SQLTrace
	)
)
ADD TARGET package0.event_file
(
	SET filename = 'C:\SQL\XE_Out\Deadlocks.xel',
		max_file_size = 250,
		max_rollover_files = 5
)
WITH (STARTUP_STATE=OFF)
;

Now that you have this XE session scripted out, it can be easily installed on multiple servers. If you encounter a deadlock problem, you can easily start the XE session and let it run to trap your deadlocks. They will be persisted to a file dedicated for the deadlocks. You can use my Deadlock Shredder script at https://bit.ly/ShredDL to read the deadlocks from the file and shred the deadlock XML into a tabular output.

Note that the default system_health XE session also captures deadlocks. I like to have a dedicated session for just deadlocks. As lightweight as XE is, sometimes it may benefit a server to turn off the system_health session. Additionally, Jonathan Kehayias has a script that will take a running trace and completely script out an XE session for it. This script can be found at https://www.sqlskills.com/blogs/jonathan/converting-sql-trace-to-extended-events-in-sql-server-2012/. Even though this script is available, I like to figure things out for myself so that I can learn what is actually going on.

Did you catch the note above where I mentioned that the default system_health XE captures deadlocks? This means that if you are still enabling trace flags 1204 and/or 1222 to capture deadlock information in your error logs, you don’t need to do that anymore. Furthermore, by using this Deadlock XE, you can have the deadlocks persisted to a file where, in combination with my Deadlock Shredder script, it will be even easier to analyze the deadlocks than trying to figure it out from the captured information in the error logs.

I hope that this post will help someone with the conversion from using SQL Profiler to using XE instead. The process for converting other traces would be the same, so it can be easily adapted. I also hope that these deadlock scripts will be helpful to you!

In my next post, I will be comparing the results of the deadlock trace and the deadlock XE session.

And Jes – thanks for hosting this month! I’m really looking forward to seeing the roundup for this month.

The post Weaning yourself off of SQL Profiler (Part 1) appeared first on SQL Solutions Group.

]]> https://sqlsolutionsgroup.com/wean-off-sql-profiler-part-1/feed/ 2 Using Linked Servers the Right Way https://sqlsolutionsgroup.com/using-linked-servers-the-right-way/ https://sqlsolutionsgroup.com/using-linked-servers-the-right-way/#comments Sun, 28 Oct 2012 17:36:16 +0000 http://sqlsolutionsgroup.com/blog/?p=181 Linked Servers are a very popular way to access distributed data in environments with lots of SQL Server instances on different serves, or even on other Database Management Systems like Oracle or MySQL. While linked servers can be very useful if used correctly, I have found that more often than not, they are not used correctly […]

The post Using Linked Servers the Right Way appeared first on SQL Solutions Group.

]]> Linked Servers are a very popular way to access distributed data in environments with lots of SQL Server instances on different serves, or even on other Database Management Systems like Oracle or MySQL. While linked servers can be very useful if used correctly, I have found that more often than not, they are not used correctly and are a source of lots of performance problems.  In this article we’ll take a look at using linked servers the right way (and a few wrong ones).

Linked Server Review

To start with, lets look at how a linked server works.  We configure a linked server by specifying the remote data source and a name for the sever.

linked1

This allows us to use the linked server name ADVENTUREWORKS to access the AdventureWorks database on the instance RKLAPTOPSS.  Note that the Linked Server name can be whatever we want it to.  It does not have to be the name of the instance. I prefer this practice because you can use the same linked server name in multiple environments (Development, QA, Production, etc.)

Authentication settings have to be configured as well.

image

In this case, I am saying to just pass through the login’s security context to the remote server.  I could also choose to always use a certain security context or to map local login’s to remote logins. The security implications of these choices are beyond the scope of this post but the most important thing is that in most cases Options 2 and 4 should not be used.

With the Remote DBA managed service from

SQL Solutions Group, we do your database dirty

work, freeing up your team for projects

that impact the bottom line.

Learn More

Four Part Naming in Linked Servers

Now that we have a linked server set up, we can access it from the local SQL Server instance.  The most common way to do this is to use four part names to refer to objects on the remote server.

<LinkedServerName>.<DatabaseName>.<Schema>.<Object>

SELECT *
FROM ADVENTUREWORKS.AdventureWorks.HumanResources.Employee
WHERE Gender = 'M'

While this is the most popular way to use Linked Servers, it is also the worst way in terms of performance.  To understand this, you need to always ask the following question: “Where will the query be optimized?”

In this case, the optimizer on the local server will determine the execution plan.  But it doesn’t know anything about indexes and statistics that exist on the Employee table on the remote server.  There may well be an index on Gender which could be taken advantage of, but the local optimizer has no way to know that.   So this query will result in a table scan, every time, regardless of remote indexes. In addition, all of the data will be sent across the network only to be filtered for Gender on the local system.

OPENQUERY()

A better way to do this is to use a function like OPENQUERY().  Essentially what we are doing is executing a remote query on the linked server and just asking for the results back.

SELECT *
FROM OPENQUERY
   (
    ADVENTUREWORKS,
    'SELECT * FROM AdventureWorks.HumanResources.Employee WHERE Gender = ''M'''
   )

The function takes two parameters.  The remote server to execute the query on, and the query itself.  Note that this is remotely executing so if the server was to another DBMS, you’d want the query written in that engine’s flavor of SQL.

This is a simple example, but imagine if the remote table had a million plus rows.  The inefficient table scan plus bringing all that over the network so that it can be filtered locally.  I have “fixed” queries that were taking 12+ hours to run and reduced the time to just minutes by making this simple change to the code.

The post Using Linked Servers the Right Way appeared first on SQL Solutions Group.

]]> https://sqlsolutionsgroup.com/using-linked-servers-the-right-way/feed/ 12