Introduction

This post describes how to use PowerShell to get your favorite toolbelt into a new database environment, your new jump box or VM that is based on a Window Server OS, without using an internet browser.

For me personally, I like to have the same tools everywhere I go. Tools (I often call it a toolbelt) that I know how to use—tools that are making me more productive/efficient, and preferably tools that have a price tag of free.

My favorite toolbelt for SQL Server consists of the following software packages:

• Microsoft Edge Chromium (replaces IE and smooths your interaction with the Web and docs.microsoft.com)
• SQL Server Management Studio (enables connectivity to SQL Server instance)
• RedGate SQL Search (finds objects or keywords inside procedure and/jobs)
• SentryOne Plan Explorer (gives more in-depth information on a query execution plan)
• ApexSQL Refactor (makes your code more readable)

Problem

While I don’t know enough about Linux, I do know enough about Windows. Unfortunately, Windows Server is still shipped with an Internet Explorer (a.k.a. IE), making it the only browser available to you. Even more unfortunate, IE is nothing but a plain pain (a.k.a. PP) to use. IE doesn’t make anything easy, especially when it’s installed on a server with a corporate group policy (a.k.a. GP) that is making downloads extremely painful.

Try to download anything external to a Windows Server with Internet Explorer and you will see a high level of PP. It’s like this browser is trying to keep you insulated from modern progress by reminding you (Fig #1) what you are not allowed to experience due to “some” security! 

Solution

While you can’t control which internet browser Microsoft ships with Windows Server and it’s a bit difficult to change GP, you completely skip using IE for downloads – introducing PowerShell (a.k.a. PS).

PS cmdlets will let you download anything that you want (your corporate security policies still apply here). This is how to use PowerShell to add your toolbelt to a new environment, easy-peasy. You don’t need to know much PS syntax and virtually all Windows Servers are shipped with PowerShell, so let’s see how easy it really is.

All you need is to incorporate the following code into a ps1 file and run it. You can use Windows PowerShell ISE (see Fig #2) or VS Code or alternatively you can run line by line from a PowerShell Command Prompt (see Fig #3).

Source code

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$client = new-object System.Net.WebClient
 
#test download datasteve image
#$client.DownloadFile(“https://steverezhener.files.wordpress.com/2019/12/datasteve-sm.jpg?w=107”,“C:\Users\Steve.Rezhener\Pictures\datasteve.jpg”)
 
#download Edge Chromium
$client.DownloadFile(“https://dl.delivery.mp.microsoft.com/filestreamingservice/files/789ec525-0feb-4d62-a24b-99fd15d117e4/MicrosoftEdgeEnterpriseX64.msi”,“C:\Users\Steve.Rezhener\Downloads\MicrosoftEdgeEnterpriseX64.msi”)
 
#download SSMS
$client.DownloadFile(“https://aka.ms/ssmsfullsetup”,“C:\Users\Steve.Rezhener\Downloads\ssms-setup-enu.exe”)
 
#download Red Gate SQL Search
$client.DownloadFile(“https://download.red-gate.com/SQL_Search.exe”,“C:\Users\Steve.Rezhener\Downloads\SQL_Search.exe”)
 
#download SentryOne Plan Explorer
$client.DownloadFile(“https://downloads.sentryone.com/downloads/sqlsentryplanexplorer/x64/PlanExplorerInstaller.exe”,“C:\Users\Steve.Rezhener\Downloads\PlanExplorerInstaller.exe”)
 
#download ApexSQL Refactor
$client.DownloadFile(“https://www.apexsql.com/zips/ApexSQLRefactor.exe”,“C:\Users\Steve.Rezhener\Downloads\ApexSQLRefactor.exe”)

If you are using Windows PowerShell ISE, just click the play button, wait for about 30 seconds and enjoy my toolbelt (see Fig #4) or feel free to change my code to download anything else you need and bypass the IE.

Enjoy installing and using the software.

Disclaimer

This blog post is partially based on https://www.thewindowsclub.com/download-file-using-windows-powershell

The post How to use PowerShell to get your toolbelt into a Windows Server without a browser appeared first on SQL Solutions Group.

Introduction

This blog post discusses how to leverage SentryOne’s SQL Sentry product to monitor database backups and integrity checks.

At SSG, we are big fans of SentryOne and have been a partner for many years. If you are not familiar with the product, it is a Performance Monitoring solution for SQL Server, as well as Analysis Services, Azure SQL Database, SQL Server on Amazon RDS and more. 

That being said, this blog post is not about why SentryOne is a great product, but rather provides a solution to a problem using custom advisory conditions. 

Problem

One of the basic responsibilities of any DBA is making sure that backups and database integrity checks are in-place. While we can monitor jobs that run these for failures, a more sure way is to get the SQL Server metadata about the last backup and last checkdb. In some cases, backups are running through third party software or from a centralized system, so a job would not even exist. And what we really want to know is “when was the last successful backup.” The same is true for database integrity checks.

Without SentryOne, we have a couple of options:

• Create a job using the dbatools PowerShell module (which we are also big fans of) and use the cmdlets Get-DbaDbBackupHistory  and Get-DbaLastGoodCheckDb for integrity checks. 
• Query msdb.dbo.backupset table (taking latest backup_finish_date column) for backup history and DATABASEPROPERTYEX (‘dbname’, ‘LastGoodCheckDbTime’) for integrity checks.  

Needless to say, those two options are both a bit cumbersome. Additionally, we would need to store this data, so we can see history.

Solution

While SentryOne comes with more than 85 built-in advisory conditions, it does not include these two items. The good news is, it’s super easy to add your own conditions.

Advisory Conditions allow us to create conditions using queries against the SentryOne repository as well as against other database(s) and several other options. Since SentryOne is already collecting the information we need for these two conditions, we can just query the repository database. 

The following query will be used to pull the required data from the repository:

Source code

SELECT esc.objectname + '.' + db.name AS server_database
, DATEDIFF(DAY,MAX(LastDbccCheckDbTime),GETDATE()) AS LastDbccCheckinDays
, DATEDIFF(DAY,MAX(LastBackupTime), GETDATE()) AS LastBackupinDays
FROM
    Device
    JOIN EventSourceConnection esc ON
        esc.DeviceID = Device.ID
    JOIN PerformanceAnalysisSqlDatabase db ON
        db.EventSourceConnectionID = esc.ID
    JOIN PerformanceAnalysisSqlFile f ON
        f.EventSourceConnectionID = esc.ID
WHERE 1=1
AND f.lastbackuptype = 'D'
and db.name NOT IN ('tempdb','distribution')
GROUP BY esc.objectname
, db.name

The following steps outline how to add this query as a custom condition across all monitored servers:

From the  Navigator, expand the Advisory Conditions node and open the Condition List, then click on Create Advisory Condition. 

1. Add a name (Track missing full database backups/Track missing database checkdbs)
2. Click on +
3. Select Add Numeric Comparison
4. Select Repository Query
5. Add the query
6. Select Any
7. Select Is Greater than
8. Select Explicit Value
9. Type 7 (or whatever threshold you want to set).
10. Save the condition

Alternatively, you can just download the two files from the following github public repository and import them: 

Custom Conditions

Your condition list should get updated once you download and import the conditions. At this point you can configure whatever actions you want as  you would for any other condition in SentryOne. These steps should make it easy to monitor database backups using SentryOne.

Disclaimer:

This blog post is partially based on the following resources:

https://www.sentryone.com/blog/query-database-growth-over-time-from-your-sentryone-sql-sentry-database

https://www.mssqltips.com/sqlservertip/1601/script-to-retrieve-sql-server-database-backup-history-and-no-backups/

https://thomaslarock.com/2018/05/using-databasepropertyex-to-find-last-good-dbcc-checkdb-time/.

The post How to monitor database backups and integrity with SentryOne SQL Sentry custom advisory conditions appeared first on SQL Solutions Group.

Using Extended Event Session, once we add filters, we can see specifics – queries that are hitting a specific database on Server A (primary) from a specific server (and many excluding the monitoring software).

Introduction

The Always On Availability Group (aka AG) feature in SQL Server provides a great HADR (high-availability and disaster recovery) solution. This HADR solution can also be used to alleviate the pain from a single database server. AG allows the user to add new servers to offload reporting needs (read replicas) without any prior domain knowledge.

To start with a new read replica, you would need a new Database Server and AG as a data synchronization mechanism (synchronous mode for time sensitive needs and asynchronous mode for everything else). Read replica provides a database on a new server that will help to lift the pressure from a single Server A (AG primary) by making Server B (AG secondary/read replica) available for all the read queries.

Problem

AG is relying on a connection string directive called Read-Intent only which instructs SQL Server to redirect all the calls to a read replica. Unfortunately, all the existing connections were defined before AG introduction, so what do we do now? How would we identify all the read queries that need a connection string change? Again, without changing connection strings, all the existing queries will still be hitting Server A, hence AG by itself would not solve the problem.

Solution

Introducing Extended Event sessions (aka XE session). XE is essentially a replacement for SQL Profiler, which enables us to see everything that is hitting the SQL Server while offering better interface and scripting capabilities. Once we add filters, we can see specifics – queries that are hitting a specific database on Server A (primary) from a specific server (and many excluding the monitoring software).

The following script creates a new XE session that ignores system queries, includes queries that are going against a Database called Database from a server called DW while excluding queries from SentryOne monitoring software.

Source code

CREATE EVENT SESSION [DW going against PRIMARY] ON SERVER 
ADD EVENT sqlserver.sql_batch_starting(
    ACTION(sqlserver.client_app_name,sqlserver.client_hostname,sqlserver.database_name,sqlserver.nt_username,sqlserver.sql_text,sqlserver.username)
 
WHERE ([package0].[equal_boolean]([sqlserver].[is_system],(0)) 
 
AND [sqlserver].[like_i_sql_unicode_string]([sqlserver].[sql_text],N'%OLTP_Database%') 
 
AND [sqlserver].[equal_i_sql_unicode_string]([sqlserver].[client_hostname],N'DW') 
 
AND [sqlserver].[not_equal_i_sql_unicode_string]([sqlserver].[client_app_name],N'SentryOne 20.0-Server')))
 
WITH (MAX_MEMORY=4096 KB,EVENT_RETENTION_MODE=ALLOW_SINGLE_EVENT_LOSS,MAX_DISPATCH_LATENCY=5 SECONDS,MAX_EVENT_SIZE=0 KB,MEMORY_PARTITION_MODE=PER_CPU
,TRACK_CAUSALITY=ON,STARTUP_STATE=OFF)
GO

Disclaimer

This blog post is partially based on the following sources:

https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/always-on-availability-groups-sql-server

https://docs.microsoft.com/en-us/sql/relational-databases/extended-events/sql-server-extended-events-sessions

The post Using Extended Event Session (XE) to track queries going against Always On AG primary appeared first on SQL Solutions Group.

This blog post is going to discuss inspection of Power BI performance when you use Power BI Gateway to push data to PowerBI.com service.

I am going to assume here that you are not new to PowerBI. I am also going to assume that you skipped PowerBI Report Server (which requires a hefty Software Assurance $$$) and settled with a more cost-effective way to run your pbix files using PowerBI.com service ($10 for Power BI Pro or $5K+ for PowerBI Premium). My last assumption would be that you are using Power BI Gateway to push your data to PowerBI.com service.

Problem

Customer: Please allow me to see Execution Logs for Power BI.

Me: Looks like you are using a Power BI Gateway. Let me find it.

Let’s start with how we normally visualize performance: We would look for data collected in a run-time. On-premise solutions like IIS would have log files and reporting platform like SSRS would have some database records that we can review. While I am very familiar with ingesting and analyzing IIS log files and building reports on SSRS by exposing the data collected in ExecutionLog3 view under a ReportServer database, I was very skeptical that Microsoft will expose anything for PowerBI.com. After all, PowerBI.com is Microsoft internal infrastructure.

I guess I was proven wrong. Of course Microsoft is not showing us how bad PowerBI was hammered to process the report or how much time it took to render a report, but Microsoft still collects enough data to weed out the worst performing queries.

Solution

Ironically, not only this is possible, but Microsoft went an extra mile and created Performance BI template for Power BI performance. This is quite awesome.

So what do we need to do to inspect PowerBI performance? We would need to:

1. Enable performance logging by setting both QueryExecutionReportOn and SystemCounterReportOn to True inside GatewayCore.dll.config xml file (you are going to need write/read permission to make changes in C:\Program Files\On-premises data gateway\ folder)

2. Set ReportFilePath variable to a shared folder that is more accessible and better managed than a local C drive (Power BI Gateway default)

3. Restart Power BI Gateway service (or upgrade if possible)

4. Download the Gateway Performance PBI template

5. Import previously downloaded template (Select File > Import > Power BI template from within Power BI Desktop)

6. Change folder path to point to log files (if applicable)

7. Enjoy it

8. Enable additional logging inside On-premises data gateway (if more logging is required)

Disclaimer

This blog post is partially based on the following Microsoft article – https://docs.microsoft.com/en-us/data-integration/gateway/service-gateway-performance.

The post Inspecting Power BI performance appeared first on SQL Solutions Group.

This blog post addresses SSRS on Always On and RSExecRole. To make sure I’m staying laser focused on this topic, I need to cover some fundamentals.

SSRS installation (SQL Server 2017 and later)

Starting from SQL Server 2017, Microsoft decided to separate SSRS from a SQL Server installation container (ISO). Keeping the pros and cons of this change aside, that means that Microsoft decided some customers would not need SSRS, and adding a separate step to install it would effectively block an accidental “next-next” type of installation. If a separate SSRS installer is triggered, it will add all the necessary SSRS plumbing to SQL Server database engine, such as ReportServer and ReportServerTemp databases to support SSRS, database roles. It will also add many other pieces that are not related to SQL Server, such as web service, web application, etc…

https://docs.microsoft.com/en-us/sql/reporting-services/install-windows/install-reporting-services?view=sql-server-ver15

Always On Availability Groups (a.k.a. AG)

Always On Availability Groups is the latest and greatest Microsoft HADR (High Availability and Disaster Recovery) solution to keep SQL Server running in case of failure. AlwaysOn Availability groups allows to enroll databases into a group that would be kept in sync between the active server (a.k.a. primary, supporting both read and write) database and passive (a.k.a. replica, supporting read only) databases.

https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/always-on-availability-groups-sql-server?view=sql-server-ver15

SSRS Subscriptions

SSRS subscription is a saved configuration that is designed to simplify report rendering and delivery. You choose a report, format, schedule and a method of delivery, and suddenly your users are more self sufficient and you have more free time.

https://docs.microsoft.com/en-us/sql/reporting-services/subscriptions/subscriptions-and-delivery-reporting-services?view=sql-server-ver15

Problem

Once you have mission critical databases included in availability groups it makes a total sense to add both ReportServer (storage for datasources, datasets, reports, etc…) and ReportServerTempDB databases (TempDB for SSRS)  as well. This way in case of a disaster recovery, your data-sources (OLTP databases for example) and reporting platform (SSRS) are up and running at the same time without any manual steps.

With Always On Availability Groups (a.k.a. AlwaysOn AG), both the data and schema are constantly (synchronous mode) or almost constantly (asynchronous mode) in-sync, but there is one user requirement. Always On requires same logins (stored in master that is not part of AG) and SQL Server Agent jobs (stored in msdb that is not part of AG). While it might look like that adding ReportServer and ReportServerTemp databases to AG would be enough to have SSRS fully supported for HADR, unfortunately that is not entirely true.

With both SSRS databases on AG, most of the functionality is available, so SSRS Report Portal will serve reports (records from the datasource, datasets, and catalog tables) and obey the security mechanism (users, policies, role tables), but SSRS subscription will suddenly stop working (despite the subscription table being in-sync). Any existing SSRS subscriptions will stop working regardless if SSRS GUID job (unfortunately, Microsoft never got to make SSRS jobs user-friendly) is enabled or not. Trying to create a new SSRS subscription will be greeted with a very cryptic error message (Fig #1).

Little known fact about SSRS: Outside of ReportServer database, SSRS is not only dependent on jobs and logins, but also on a custom database role called RSExecRole that is created by the SSRS installer. This role needs to exist in both master and msdb and since both of their databases are normally not in AG (per server and not in-sync) and SSRS is not actually installed on the replica, SSRS subscriptions would no longer work.

Solution

The solution is fairly easy and will involve following Microsoft documentation that is designed for a scenario of moving SSRS from one server to another.

https://docs.microsoft.com/en-us/sql/reporting-services/security/create-the-rsexecrole?view=sql-server-ver15ecrole?view=sql-server-ver15

The following code will create RSExecRole database role in both master and msdb databases and will grant all the necessary permissions needed for SSRS subscription functionality.

Source code

USE master;
GO
IF NOT EXISTS (SELECT 1 FROM sys.database_principals WHERE [type] = 'R' AND [name] = 'RSExecRole') BEGIN
    CREATE ROLE [RSExecRole];
END
GRANT EXECUTE ON dbo.xp_sqlagent_enum_jobs TO [RSExecRole];
GRANT EXECUTE ON dbo.xp_sqlagent_is_starting TO [RSExecRole];
GRANT EXECUTE ON dbo.xp_sqlagent_notify TO [RSExecRole];
GO
USE msdb;
GO
IF NOT EXISTS (SELECT 1 FROM sys.database_principals WHERE [type] = 'R' AND [name] = 'RSExecRole') BEGIN
    CREATE ROLE [RSExecRole];
END
GRANT EXECUTE ON dbo.SP_ADD_CATEGORY TO [RSExecRole];
GRANT EXECUTE ON dbo.SP_ADD_JOB TO [RSExecRole];
GRANT EXECUTE ON dbo.SP_ADD_JOBSCHEDULE TO [RSExecRole];
GRANT EXECUTE ON dbo.SP_ADD_JOBSERVER TO [RSExecRole];
GRANT EXECUTE ON dbo.SP_ADD_JOBSTEP TO [RSExecRole];
GRANT EXECUTE ON dbo.SP_DELETE_JOB TO [RSExecRole];
GRANT EXECUTE ON dbo.SP_HELP_CATEGORY TO [RSExecRole];
GRANT EXECUTE ON dbo.SP_HELP_JOB TO [RSExecRole];
GRANT EXECUTE ON dbo.SP_HELP_JOBSCHEDULE TO [RSExecRole];
GRANT EXECUTE ON dbo.sp_verify_job_identifiers TO [RSExecRole];
GRANT SELECT ON dbo.syscategories TO [RSExecRole];
GRANT SELECT ON dbo.sysjobs TO [RSExecRole];
GO

Of course it would be nice if RSExecRole database role and permissions were created by default, regardless of the SSRS installation, by making it part of the “system” objects. That would be a small price to pay for a happier DBA. Wouldn’t it be nice? As the Beach Boys would sing it, “you know its gonna make it much better”!

The post How to ensure that all the SSRS features are working while using Always On Availability Groups OR have an RSExecRole as a default appeared first on SQL Solutions Group.

One of the all-time best performing posts on our blog is from Randy Knight and outlines how to use Linked Servers the right way. In this post from Steve Rezhener, he shows how OPENQUERY can make all the difference with Linked Servers.

Introduction (the good)

If you are not familiar with a linked server concept in SQL Server, don’t worry, you probably haven’t needed it yet. You are fortunate to source all your data needs from a single database server, and that is fine. Or maybe you are consuming your SQL Server needs from Azure Single Database or AWS RDS (both public cloud solutions don’t support linked server out of the box).

Most likely this is going to change (Azure VM and AWS EC2 have full support) and you will have to join data between multiple database servers, or even between different RDBMS systems. For example: All the transactional sales data is stored in SQL Server, but all the analytical sales data is stored on another SQL Server server (this could even be MySQL).

This is where Linked Server comes in handy, especially at the data discovery phase. When building a prototype needs to happen quickly and there is no time to adhere to the best practices, Linked Server can be the answer.

This blog post outlines the pros and cons of linked servers in SQL Server as well as how to reduce the cons of using an OPENQUERY.

Linked Servers Basics

According to Microsoft, linked servers enable the SQL Server Database Engine and Azure SQL Database Managed Instance to read data from the remote data sources and execute commands against the remote database servers (for example, OLE DB data sources) outside of the instance of SQL Server. 

Linked Server is a pure convenience feature. With very little to no code changes, you can suddenly join multiple tables between local and remote/cloud servers. For example, let’s use WideWorldImporters transactional database

Linked Server effectively removes the need for a solution that will move and synchronize the data. You just setup a Linked Server (see Fig #2) and add a Linked Server reference in front of the three-part naming convention (see Fig #3). That’s all. A small price to pay for lots of convenience.

 

Problem (the bad)

While the Linked Server feature makes it easy to join tables between two or more different servers, it’s not free and it comes with a price tag: performance overhead. While joining a few small tables might not add noticeable pressure to the server, joining 3-5 fairly large remote tables might introduce locking and blocking and increase the run-time from seconds to minutes.

The main problem occurs when you run queries against a remote server, which is not healthy. When SQL Server runs a query with Linked Server, it will use the least optimal execution plan due to lack of knowledge of those remote tables. Meaning, your local SQL Server is clueless on remote table indexes and statistics, so it might use an incorrect joining mechanism and might be grossly inefficient.

For example, if you had to join all January 2013 orders between OLTP and OLAP tables and compare revenue per product while showing the five top contributors, we might build the following query (see Fig #4) to achieve that goal:

Reviewing Statistics IO (see Fig #5) and execution plan details (see Fig #6), we see:

 

As you might notice (far right), 93% of the query execution cost goes to a “mysterious” remote query.

Solution (a way to make it perfect)

One of the easiest ways to fix performance problems with a linked server is to run it via OPENQUERY.

What is OPENQUERY? According to Microsoft, OPENQUERY executes the specified pass-through query on the specified linked server. This server is an OLE DB data source. We can reference OPENQUERY in the FROM clause of a query as if it were a table name. We can also reference OPENQUERY as the target table of an INSERT, UPDATE, or DELETE statement. This is subject to the capabilities of the OLE DB provider. Although the query may return multiple result sets, OPENQUERY returns only the first one. 

One of the main advantages of OPENQUERY is remote execution. This  means the local server sends the query to the remote server with knowledge of those remote tables that are now local to the query. By the way, remote execution also enables the use of native syntax of the remote server, so you can take advantage of other RDBMS system performance tricks.

Here is how our original query will look with an OPENQUERY (see Fig #7). It’s still using the same linked server, but it happens indirectly now with an  OPENQUERY.

When reviewing Statistics IO (see Fig #8) and execution plan details (see Fig #9), we now see: 

Let’s compare Statistics IO and execution plans to see the differences between direct query linked server usage vs. an OPENQUERY linked server.

The most noticeable differences are:

1. WorkFile reads in Statistics IO (see Fig #10)
2. Remote Query 93% cost vs. most distributed cost in Execution Plan (see Fig #11) and partially replaced with Remote Scans.

Why Does It Work?

The main reason OPENQUERY will usually perform better is knowledge of the table in terms of indexes and stats, knowledge that a direct linked server doesn’t have.

As you can see, we have managed to cut the run-time from 22 seconds (using direct link server) down to 2 seconds (using OPENQUERY). Additionally, we can add indexes specific to the query to make OPENQUERY even faster.

Disclaimer

Since nothing is certain in this life other than taxes and death, your mileage may vary (“YMMV”). Please test it (preferably using DBCC FREEPROCCACHE) prior to deployment. In addition the query above that reduced run-time by 11 times, I also scored a drastic performance gain by using OPENQUERY in one of the clients. In that example, replacing all three direct linked server references reduced run-time by 10 times, to just a few seconds.

The only caveat with OPENQUERY, since it can’t accept parameters, are the use of dynamic SQL and “fighting the quotes” – making sure to escape the single quotes that are internally used by a SQL Server to identify literal, i.e. text.

The post Linked Server in SQL Server: the good, the bad, and the way to make it perfect! appeared first on SQL Solutions Group.

Data Governance plays an important role in building a successful data-driven company as it helps business users to trust the data. While a typical Data Analytics project starts with identifying all relevant transactional sources of information, building your Data Warehouse according to business needs and feeding (and converting) the data as often as possible, one can argue that without operational reports alerting on a lag between the two systems, your Data Analytics function is doomed. No matter how great your Data Warehouse is, if the users are constantly complaining about bad data, your Data Warehouse project days are numbered.

I am assuming that you followed best practices and isolated your company analytical server (OLAP) from your transactional server (OLTP). You also set up an ongoing replication in near real-time. As much as business users are happy to analyze all the historical data (great for trending analysis that requires volume of data), they would also love to see real-time data (great for an e-commerce company that is coming up with new products). But how do you ensure your dependent tables are in-sync?

Problem

How do you compare data between source and destination tables using a repeatable process? One of the first reporting needs in a Data Governance report is comparing counts to identify any missing records. Having a code that allows to get a record count from any table anywhere (local/remote server) will make it easy.

Solution

We are going to build a reusable stored procedure that could help in a Data Governance Report to capture record counts between source and destination tables to compare and keep track of trending.

While the first challenge is building the flexibility that allows to query a table/view record count on any server, the second challenge is how to avoid resources hungry COUNT(*) that could lead to performance issues.

The following stored procedure does just that – capturing a record count virtually anywhere using sp_executesql with dynamic SQL and taking advantage of statistical record count from sys.dm_db_partition_stats to avoid performance problems.

Source code

USE [yourdatabasename]
 GO
 /****** Object:  StoredProcedure [dbo].[usp_GET_RecordCountfromAnyTable]    Script Date: 4/23/2019 3:41:40 PM ******/
 SET ANSI_NULLS ON
 GO
 SET QUOTED_IDENTIFIER ON
 GO
 -- =============================================
 -- Author: Steve Rezhener
 -- Create date: 04/23/2019
 -- Description: Getting record count from any table anywhere (local/remote server)
 -- EXEC dbo.usp_GET_RecordCountfromAnyTable @TagName = 'AWS EC2', @ServerName= NULL, @DatabaseName='yourdatabase', @SchemaAndTableName='yourtablename'
 
CREATE PROCEDURE [dbo].[usp_GET_RecordCountfromAnyTable]
@TagName AS VARCHAR(20)
,@ServerName AS VARCHAR(100)
,@DatabaseName AS VARCHAR(50)
,@SchemaAndTableName AS VARCHAR(100)
 
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
 
DECLARE @SQLString1 AS NVARCHAR(MAX)
DECLARE @SQLString2 AS NVARCHAR(MAX)
DECLARE @ObjectID AS BIGINT
DECLARE @ParmDefinition NVARCHAR(500);
DECLARE @SchemaName AS VARCHAR(50) = PARSENAME(@SchemaAndTableName,2)
DECLARE @TableName AS VARCHAR(50) = PARSENAME(@SchemaAndTableName,1)
 
 
	IF ISNULL(@ServerName,@@SERVERNAME) = @@SERVERNAME -- local
	BEGIN
 
		IF @ServerName IS NULL SET @ServerName = @@SERVERNAME
 
		SET @SQLString1 = 'SELECT @ObjectIDOUT = tabs.OBJECT_ID
		FROM ' + @DatabaseName + '.sys.tables AS tabs
		INNER JOIN ' + @DatabaseName + '.sys.schemas AS mas ON tabs.schema_id = mas.schema_id
		WHERE 1=1
		AND tabs.name = ''' + @TableName + '''
		AND mas.name = ''' + @SchemaName + '''
		'
 
		PRINT '@SQLString1: ' + @SQLString1
		SET @ParmDefinition = N'@ObjectIDOUT BIGINT OUTPUT';
		EXECUTE SP_EXECUTESQL @SQLString1, @ParmDefinition, @ObjectIDOUT = @ObjectID OUTPUT
		PRINT '@ObjectID: ' + CONVERT(VARCHAR,@ObjectID)
 
		SET @SQLString2 = N'SELECT ''' + @TagName + ''' AS TagName, ''' + @ServerName + ''' AS ServerName, ''' + @DatabaseName + ''' AS DatabaseName, ''' + @SchemaAndTableName + ''' AS SchemaAndTableName, row_count AS RecCount
		FROM ' + @DatabaseName + '.sys.dm_db_partition_stats
		WHERE 1=1 AND object_id = ' + CONVERT(VARCHAR,@ObjectID) + ' AND index_id=1'
 
 
		PRINT '@SQLString2: ' + @SQLString2
		EXECUTE SP_EXECUTESQL @SQLString2
 
	END
	ELSE -- remote server via linked server
	BEGIN
 
		SET @SQLString1 = 'SELECT @ObjectIDOUT = tabs.OBJECT_ID
		FROM [' + @ServerName + '].' + @DatabaseName + '.sys.tables AS tabs
		INNER JOIN [' + @ServerName + '].' + @DatabaseName + '.sys.schemas AS mas ON tabs.schema_id = mas.schema_id
		WHERE 1=1
		AND tabs.name = ''' + @TableName + '''
		AND mas.name = ''' + @SchemaName + '''
		'
 
		PRINT '@SQLString1: ' + @SQLString1
		SET @ParmDefinition = N'@ObjectIDOUT BIGINT OUTPUT';
		EXECUTE SP_EXECUTESQL @SQLString1, @ParmDefinition, @ObjectIDOUT = @ObjectID OUTPUT
		PRINT '@ObjectID' + CONVERT(VARCHAR,@ObjectID)
 
 
		SET @SQLString2 = N'SELECT ''' + @TagName + ''' AS TagName, ''' + @ServerName + ''' AS ServerName, ''' + @DatabaseName + ''' AS DatabaseName, ''' + @SchemaAndTableName + ''' AS SchemaAndTableName, row_count AS RecCount
		FROM [' + @ServerName + '].' + @DatabaseName + '.sys.dm_db_partition_stats
		WHERE 1=1 AND object_id = ' + CONVERT(VARCHAR,@ObjectID) + ' AND index_id=1'
		--PRINT '@SQLString2: ' + @SQLString2
 
 
		PRINT '@SQLString2: ' + @SQLString2
		EXECUTE SP_EXECUTESQL @SQLString2
 
	END
 
END

Equipped with this stored procedure, it is just a matter of time that you can build an operational report to look sharp and shinny that you can share with your business users, as you can see in this image: 

The post Data Governance – your ultimate stored procedure to get a record count from any table and compare appeared first on SQL Solutions Group.

The SSISDB database (a.k.a. the Integration Services catalog) was introduced back in SQL Server 2012 as a way to de-clutter MSDB database and provide some built-in logging and reporting infrastructure. In a nutshell, SSISDB is an SSIS framework (see Fig #1) making SQL Server Integration Services more robust and enterprise-friendly by providing the following features:

• Database backup
• Database encryption
• Support for environments
• Support project and package parameters
• Package versioning
• Custom SSRS performance reports built into SSMS
• Support for deployment within SSDT

While overall SSISDB Catalog was and still is a great enhancement to SQL Server Integration Services, some previously working fine feature took a wrong turn. This blog post is going to cover the “behavioral” changes that Microsoft made to SQL Server Agent job notification as part of the SSISDB Catalog, specifically the error message reporting on failed package.

Problem

Prior to SQL Server 2012, an email notification on a failed SQL Server Agent job step with an SSIS package was really informative. The error message was specific enough, as it was providing the error from the SSIS execution engine. When Microsoft introduced SQL Server 2012, it decided to replace this specific SQL Server Agent job email notification with a generic message directing the user to the SSIS Catalog error log (see Fig #2).

I’m assuming when most of us are receiving Operational emails, we want to know not only if to react or not, but also how to react. This is especially important now, since we use mobile devices to read emails and might not have access to the a real computer.

Unfortunately, the email above gives only an indication that some SSIS package failed without giving any details. For example, in the past (prior to SQL Server 2012), you would know that your package failed due to connectivity issue by just reading an email and decide to ignore the notification. Now you don’t have that option. You would need a real computer to fire-up SSMS and run a custom SSRS on that package, adding many clicks to see the details error. Needless to say, running to your desktop/laptop to open SSMS only to discover connectivity problems is unnecessary overkill.

Solution: Use T-SQL to create a better SQL Server Agent job email notification 

The solution will involve a T-SQL query (I should probably make it a stored procedure at some point) that would connect tables and records between both MSDB and SSISDB databases to bring back a detailed message on a failed SSIS package. Equipped with an email that has detailed error message, you can react without leaving your couch – fix it or just let it retry.

In a nutshell this solution is to going to:

1. Find the job name based on a current SPID
2. Find a job with a failed step
3. Connect step with SSIS package name
4. Use the SSIS package name to find run-time errors in SSISDB catalog
5. Bubble us to an inner exception to include only detailed error
6. Locate a default job operator
7. Send a detailed error report

The following script would need to be added as last step in a job (you can disable the job alert) with all the steps that include SSIS packages going to that last step on failure (Fig 3).

Additionally, the used-to-be last step before Report SSIS failure was introduced that will now precede that step will need to quit the job reporting success (Fig 4) instead of continuing to the next step..

Source code

DECLARE @Full_ssis_command VARCHAR(4000),
 @Job_step_id INT,
 @Package_name VARCHAR(4000),
 @EMailBody NVARCHAR(2000),
 @EMailSubject VARCHAR(150),
 --@Job_name VARCHAR(100),
 @Job_id UNIQUEIDENTIFIER
 
SELECT	@Job_id = Job_id
		--@Job_name = [name] --AS job_name
FROM	sys.dm_exec_sessions AS ions
	INNER join msdb.dbo.sysjobs AS jobs ON jobs.job_id
				= CAST(CONVERT(BINARY(16), SUBSTRING(ions.[program_name],CHARINDEX('(Job 0x',ions.[program_name],1) + 5 , 34),1) AS UNIQUEIDENTIFIER)
WHERE	1=1
	AND ions.session_id = @@SPID --63
 
--print @Job_name
 
SELECT TOP 1 @Job_step_id = Step_id
FROM msdb.dbo.sysjobhistory (NOLOCK)
WHERE Run_status <> 1
 AND Step_id > 0
 AND Job_id = @Job_id
ORDER BY Instance_id DESC
 
SELECT @Full_ssis_command = Command
FROM msdb.dbo.sysjobsteps (NOLOCK)
WHERE Job_id = @Job_id
 AND Step_id = @Job_step_id
 
 PRINT @Full_ssis_command
 
IF @Full_ssis_command LIKE '%.dtsx%'
 BEGIN
 SELECT @Package_name = RIGHT(LEFT(@Full_ssis_command,CHARINDEX('.dtsx', @Full_ssis_command)), CHARINDEX('\', Reverse(LEFT(@Full_ssis_command, Charindex('.dtsx', @Full_ssis_command)-1)))) + 'dtsx'
 END
 
SELECT TOP 1 @EMailBody = LEFT([Message], 2000), @EMailSubject = 'Package : ' + Package_name + ' failed ON :' + CONVERT(VARCHAR,Message_time)
--[Message_time],
-- [Extended_info_id],
-- [Package_name],
-- [Message_source_name],
-- [Subcomponent_name],
-- [Package_path],
-- [Execution_path],
-- LEFT([Message], 400)
FROM SSISDB.[catalog].[Event_messages] (NOLOCK)
WHERE [Package_name] = @Package_name
 AND Event_name = 'OnError'
 AND Message_time >= DATEADD(DD,0,DATEDIFF(DD,0,GETDATE()))
 AND Operation_id IN (SELECT Max(Operation_id)
 FROM SSISDB.[catalog].[Event_messages](NOLOCK)
 WHERE [Package_name] = @Package_name)
ORDER BY Message_time ASC
 
DECLARE @operator_name AS VARCHAR(100) = (SELECT top 1 name from msdb.dbo.sysoperators)
PRINT @operator_name
 
exec msdb.dbo.sp_notify_operator  @profile_name = NULL ,
    @id = NULL,
    @name = @operator_name,  
    @subject = @EMailSubject,  
    @body = @EMailBody

Here is the improved SQL Server Agent job email notification with a detailed message that we can expect now (Fig 5).

The post Bringing back an informative SQL Server Agent job email notification on failed SSIS packages appeared first on SQL Solutions Group.

Introduction

The SSISDB database (a.k.a. the Integration Services catalog) was introduced back in SQL Server 2012 as a way to de-clutter the MSDB database and provide some built-in logging and reporting infrastructure. In a nutshell, SSISDB is an SSIS framework making SQL Server Integration Services more robust and enterprise-friendly by providing the following features:

• Database backup
• Database encryption
• Support for environments
• Support project and package parameters
• Package versioning
• Custom SSRS performance reports built into SSMS
• Deployment directly from SSDT

Since SSISDB Catalog is going to log package run-time information, Microsoft created a SQL Server Agent job (SSIS Server Maintenance Job, see Fig .1) to do the in-house cleaning. This job is running two steps (see Fig 2) to enforce the data retention policy (based on SSISDB Catalog settings) instead of letting all the data collection activities to eat all your disk space and effectively brake your SSIS run-time service (if logging is enabled). 

While SSIS Server Operation Records Maintenance (a.k.a. step #1) is in charge of removing all the old (default=365 days) logging, SSIS Server Max Version Per Project Maintenance (a.k.a. step #2) is in change of removing all old (default=10 versions) package versioning.

Problem

“SSIS Server Maintenance Job” SQL Server Agent job started to fail  with the following error:

Executed as user: #MS_SSISServerCleanupJobLogin##. A cursor with the name ‘execution_cursor’ does not exist. [SQLSTATE 34000] (Error 16916)

This run-time error (trying to deallocate a CURSOR that doesn’t exist) happens inside Step #1 which runs a ssisdb.internal.cleanup_server_retention_window stored procedure. I’m  going to speculate and say that this bug was unintentionally introduced by adding more features (most likely, it’s the server_operation_encryption_level which was introduced with SQL Server 2012 SP2). While I first spotted this problem in SQL Server 2017 with CU12, unfortunately it’s still a problem as of December 2019 with the latest and greatest CU18 (14.0.3257.3).

Solution

You can wait for Microsoft to fix it in a next CU (a.k.a. Cumulative Update) or you can fix it yourself by modifying the code inside ssisdb.internal.cleanup_server_retention_window stored procedure.

A little warning here: You are going to be modifying the code that was created by Microsoft. You need to be comfortable with taking that risk.

You would need to alter the procedure by moving a DEALLOCATE statement from line #175 to #177 to fix this failed SSIS job (see Fig #3).

This post is based on this MSDN post.

The post How to fix a failed “SSIS Server Maintenance Job” job appeared first on SQL Solutions Group.

The SSISDB database (a.k.a. the SQL Server Integration Services catalog) was introduced back in SQL Server 2012 as a way to de-clutter the MSDB database and provide some built-in logging and reporting infrastructure. In a nutshell, SSISDB is an SSIS framework making SQL Server Integration Services more robust and enterprise-friendly by providing the following features:

• Database backup
• Database encryption
• Support for environments
• Support project and package parameters
• Package versioning
• Custom SSRS performance reports built into SSMS
• Deployment directly from SSDT

Fortunately or unfortunately, installation of a new SQL Server instance does not include the creation of the SSISDB Catalog, even if SQL Server Integration Services is included in the installation. For some reason, Microsoft decided to leave that part to the user (luckily, this is a one-time sunk cost).

Once SQL Server installation is completed, you would need to create Catalog using SSMS (see Fig #1) by expanding Management, right clicking on Integration Services Catalogs, and clicking on Create Catalog menu item (see Figure 1).

Problem

Once you are done with the full SQL Server installation and ready to create Catalog (SSISDB Catalog to be exact), you might be “greeted” with the following error – “The catalog backup file 'C:\Program Files\Microsoft SQL Server\140\DTS\Binn\SSISDBBackup.bak' could not be accessed. Make sure the database file exists, and the SQL Server service account is able to access it.(Microsoft.SqlServer.IntegrationServices.Common.ObjectModel)“.

I’m sure you were surprised as I am! What? Why would creation of SSISDB Catalog have anything to do with SSISDB database backup (see Fig #2)? Apparently it’s one of the internal steps that Microsoft does behind the scenes when you are installing SQL Server. To be absolutely specific, installation of an Integration Services feature. Strange, right? The real problem is that when you are trying to create an SSISDB Catalog, Microsoft “assumes” that there is database backup file (see Figure 2).

Solution

Apparently, if you forget to enable installation of Integration Services feature, this is the message that you are going to see. The solution involves going back to the installation (see Figure 3) and checking the Integration Services feature.

Once you have Integration Services enabled (see Figure 4) and the installation completed, you should be able to Create Catalog without any problem.

Disclaimer

This blog post is based on the following forum discussion https://www.sqlservercentral.com/forums/topic/missing-ssisdbbackup-bak.

The post Can not create Catalog (SSISDB) due to an error appeared first on SQL Solutions Group.