Both teams had already been doing the sensible thing — running UPDATE STATISTICS on the tables involved — and this sort of worked … sometimes, for a while. That “sometimes, for a while” is the tell. When refreshing statistics gives you unpredictable, temporary relief, it usually means statistics were never the real problem.

Picture a stored procedure that runs constantly — a polling job, an ETL step, a queue processor. The application hands it a batch of work as a table-valued parameter, and the proc uses that batch against a big table. Stripped down and anonymized:

-- The table type the TVP is built on
CREATE TYPE dbo.KeyList AS TABLE
(
    KeyId BIGINT PRIMARY KEY
);
GO

CREATE OR ALTER PROC dbo.usp_DoSomething
    @Work dbo.KeyList READONLY      -- some runs pass 12 rows, some pass 40,000
AS
BEGIN
    DELETE t
    FROM dbo.BigEventTable AS t
    JOIN @Work AS w ON w.KeyId = t.KeyId
    WHERE t.CompletedDate IS NOT NULL;
END;

Most of the time it runs in milliseconds. Then, seemingly at random, CPU slams to 100% and stays there, alerts fire off, and somebody connects and runs UPDATE STATISTICS (or restarts the app, or clears the cache). CPU drops. Everybody exhales. Then it happens again.

Here’s the thing: the data didn’t change in any meaningful way, the indexes are fine, and your statistics were never stale. The problem is the @Work table variable and how SQL Server estimates the number of rows in it.

Every table variable — whether you DECLARE one locally or receive it as a table-valued parameter — shares one root limitation: it carries no column-level statistics. There’s no histogram telling the optimizer how the values are distributed. It’s flying blind on everything except, at best, the raw row count. And how it estimates that count depends on which flavor you’re using.

Local table variables (DECLARE @x TABLE).

Before SQL Server 2019, the optimizer simply guessed one row, every time. Estimate one row, build a plan for one row — usually a nested loop — then watch it fall over when 40,000 rows show up. SQL Server 2019 at compatibility level 150 improved this with Table Variable Deferred Compilation: it waits until the variable is populated, then compiles using the real count. Genuine improvement — but it sets that estimate once, at first compilation, and caches the plan. It does not recompile every execution.

Table-valued parameters (@x dbo.SomeType READONLY).

This is the flavor that bit my clients. Because a TVP is materialized by the caller before the statement compiles, the optimizer has known the true row count since TVPs shipped in 2008 — they never had the one-row guess. But that accurate count is captured at the first compile and baked into the cached plan, exactly like a sniffed parameter value.

See the common thread? Whichever flavor you use, you land in the same place: a plan compiled for one row count, then reused for a wildly different one.

• The proc first compiles on a run where @Work holds 12 rows → SQL Server caches a tidy nested-loop plan.
• A later run passes 40,000 rows and reuses that cached small-batch plan.
• The nested loop seeks into dbo.BigEventTable 40,000 times, and there’s your CPU.

That also explains the maddening “UPDATE STATISTICS fixed it” experience. The table variable has no statistics to update, so the script wasn’t correcting anything the optimizer uses for @Work. What it was doing — as a side effect — is invalidating the cached plan and forcing a recompile. Land that recompile on a big-batch run and you get a good plan; land it on a small batch and you re-cache the bad one. It’s a coin flip dressed up as a fix.

To confirm it, pull the actual execution plan and look at the operator reading the table variable. If Estimated Number of Rows is wildly smaller than Actual (literally 1 on a pre-2019 local variable, or the stale first-compile count on a TVP), you’ve found it. On a busy server, Query Store will usually be flagging the same statement as a top CPU consumer with more than one plan on file.

Append a statement-level hint:

DELETE t
FROM dbo.BigEventTable AS t
JOIN @Work AS w ON w.KeyId = t.KeyId
WHERE t.CompletedDate IS NOT NULL
OPTION (RECOMPILE);

OPTION (RECOMPILE) tells SQL Server to recompile just this statement, on every execution, using the actual row count of @Work at that moment. Twelve rows tonight, forty thousand tomorrow — it builds the right plan each time. It’s the reliable, deterministic version of the recompile your UPDATE STATISTICS script was triggering by accident, and it works the same for a local table variable or a TVP.

What makes it the right first move: it’s surgical (scoped to one statement — the rest of the proc’s plan stays cached), it’s logic-neutral (you’re changing how the statement compiles, not what it does, which makes it an easy sell on vendor code), and the per-execution compile cost is trivial next to a runaway nested loop pinning every core. In both of my recent cases, CPU dropped from sustained 90%-plus back to normal within minutes — no index rebuild, no schema change, no restart.

Often, yes. A temporary table (#Work) carries real statistics — including a histogram — and participates in normal recompilation thresholds, fixing the estimate at the source. If the batch is genuinely variable and feeds joins against big tables, a #temp table is frequently the better long-term answer; a common pattern is to copy a TVP into a #temp at the top of the proc precisely so the optimizer gets real statistics. So why reach for the hint first? Because it’s the smallest possible change. When I’m putting out a fire on someone else’s code at 11 PM, “add one line” beats “refactor the procedure.” Ship the hint to stop the bleeding, then propose the temp-table refactor as the permanent fix.

• It’s per-execution. If the statement runs hundreds of times per second, the cumulative compile cost can become its own problem. For a periodic batch or polling job it’s a non-issue; for a hot OLTP path, prefer the temp-table route.
• Keep functions off your join columns. In one case the join also wrapped the key in ABS() — ON ABS(w.KeyId) = t.KeyId. A function on the column is non-SARGable and kills index seeks. Fixing the estimate solved the CPU, but that was a second sin worth cleaning up.
• Don’t confuse it with WITH RECOMPILE on the procedure. Statement-level OPTION (RECOMPILE) recompiles one statement (and can use actual parameter values too); procedure-level WITH RECOMPILE throws away the entire plan on every call.

• When UPDATE STATISTICS gives you unreliable, temporary relief, stop updating statistics and ask why a recompile helps. The answer is often a table variable or a TVP.
• Neither carries column statistics. Local variables misestimate the count (one-row guess pre-2019; cached first estimate after); TVPs get the count right but cache it from the first call. Different routes, same cliff.
• Reserve both for what they’re good at — small, predictable row counts and passing modest sets between tiers. The moment one carries thousands of rows into a join against a big table, it’s the wrong tool; copy it into a temp table.

Chasing an intermittent CPU problem that “fixes itself” and then comes right back? SQL Solutions Group has spent years running these down for clients — we’re happy to help you find the one line of code that’s lighting up your server.

The post Table Variables, TVPs, and the One-Line CPU Fix appeared first on SQL Solutions Group.

We recently had a customer where Windows logins to a SQL Server Availability Group started throwing “Cannot generate SSPI context” — but only sometimes, and only for some people. The same user, on the same workstation, would fail one minute and connect the next. Applications were fine. Nothing had changed on SQL Server.

It took the better part of two days and a lot of second-guessing to run down, and the culprit turned out to be something most of us never think about: which domain controller handed out the Kerberos ticket.

In the spirit of full disclosure, I burned most of the first day chasing the usual SPN suspects before the real cause showed itself. So let me save you that day.

Here’s what we’ll cover:

• Why a healthy SQL Server starts failing Windows auth for some users and not others
• How to prove it’s the domain controllers, not your SPNs
• The interim fix (no restart) and the permanent one

The Scenario

Two-node Always On AG, Windows authentication, and an intermittent “The target principal name is incorrect. Cannot generate SSPI context.” The pattern that makes you question your sanity:

• Same user, same machine — works, then fails, then works.
• Fails from some client machines, succeeds from others, with no obvious pattern.
• Applications connect fine; only interactive users (SSMS, Visual Studio Code) hit it.
• No failover, no deploy, no SPN changes.

Everything you’d normally check comes back clean. The SPNs are registered correctly. No duplicates. The service account is right on every node. By every test you’ve ever used, this should work — and half the time, it does.

What’s Actually Happening?

Here’s the thing: this isn’t an SPN problem at all. It’s an encryption-type problem, and it’s a delayed gift from the November 2022 Kerberos hardening updates (CVE-2022-37966 / KB5021131).

A quick refresher: When you authenticate to SQL Server over Kerberos, a domain controller hands you a ticket encrypted with the SQL service account’s key. Historically that was RC4. Microsoft has been pushing everyone to AES, and an account can hold keys for both. Which type a DC issues depends on the account’s msDS-SupportedEncryptionTypes attribute.

Before November 2022, DCs would quietly fill in RC4/AES support for any account that hadn’t declared one. That update stopped that. Now, if an account’s msDS-SupportedEncryptionTypes is null — and most service accounts have never had it set — each DC falls back to its own DefaultDomainSupportedEncTypes registry setting, which depends on that DC’s patch level and configuration.

In a shop with more than a couple of DCs patched on different schedules by different hands, that’s the trap: one DC issues AES, another issues RC4. If the account’s AES key happens to be unusable — a salt mismatch, a stale key — the AES tickets fail and the RC4 tickets work. And since a client doesn’t choose its domain controller, whether your connection succeeds is a coin flip. That is your “comes and goes.”

How to Prove It

This is the move that finally cracked it for us, and it’s the first one to reach for next time. Pick a client that fails and one that works, and ask each to fetch a ticket for the same SQL SPN:

Source code

klist purge
klist GET MSSQLSvc/sqlnode1.contoso.com
klist

On the working client:

Server: MSSQLSvc/sqlnode1.contoso.com @ CONTOSO.COM
KerbTicket Encryption Type: RSADSI RC4-HMAC(NT)
Kdc Called: DC02.contoso.com

On the failing client — same SPN, same account:

Server: MSSQLSvc/sqlnode1.contoso.com @ CONTOSO.COM
KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
Kdc Called: DC01.contoso.com

Read that twice. One client got RC4 from DC02 (works), the other got AES from DC01 (fails). The encryption type of the ticket — and the DC that issued it — is the entire story, and no SPN tool will ever show it to you.

While you’re there, confirm what each DC is doing and rule out the things it isn’t:

Source code

# account: null enctype + OLD/consistent password rules OUT a recent reset
 
'DC01','DC02' | ForEach-OBJECT {
 
Get-ADUser svc_sql -Properties msDS-SupportedEncryptionTypes,pwdLastSet -Server $_ |
 
Select-OBJECT @{n='DC';e={$_}}, msDS-SupportedEncryptionTypes
 
}
 
repadmin /replsummary # 0 failures = not a REPLICATION problem

Null enctype, a consistent (old) password, zero replication failures, and two DCs handing out different ticket types is the fingerprint.

Resolution

There are two fixes, and which you reach for depends on whether you can take a maintenance window.

The no-restart interim fix

Pin the account to RC4 so every DC issues the ticket that already works:

Source code

Set-ADUser svc_sql -REPLACE @{'msDS-SupportedEncryptionTypes'=4} # 4 = RC4 ONLY

It’s a settings-only change, it replicates in seconds, and it restores service across the board without touching SQL Server. Now, pinning to RC4 sounds like exactly the wrong direction given that RC4 is what these updates are retiring. And it is, as a permanent answer. But as a bridge, it’s the right call: it’s deterministic, it needs no downtime, and it buys you the window to do the real fix. Don’t leave it there forever, and resist the urge to set RC4-plus-AES (0x1C) — the DC will still prefer the broken AES and you’ll be right back where you started.

The permanent fix

When you can take a brief SQL restart:

1. Reset the service account password and restart the SQL Server service. This regenerates a correct AES key that the running service and the DCs agree on.
2. Set the account explicitly to AES so every DC issues the same thing: 

   Source code

   Set-ADUser svc_sql -REPLACE @{'msDS-SupportedEncryptionTypes'=24} (24 = AES128 + AES256)

3. Get your domain controllers consistent — same patch level, same DefaultDomainSupportedEncTypes. This one’s domain-wide hygiene; the same trap is waiting on your other service accounts as RC4 gets disabled.

Gotchas

A few things make this especially nasty on SQL Server:

• NTLM fallback lies to you. Many SQL connections silently fall back to NTLM, so most users “work” while Kerberos is quietly broken. Don’t trust “it connects” — check auth_scheme in sys.dm_exec_connections. The boxes that fail loudest are often locked-down admin jump boxes that can’t fall back, which makes them your honest canary.
• Connection pooling masks it. A reused pooled connection reports whoever opened it. Probe with pooling off.
• It cascades into double-hops. SSRS, linked servers, anything delegated rides on top of this. If the first hop can’t do Kerberos, no amount of delegation config saves you. Don’t go down that road until single-hop Kerberos is solid.

Lessons Learned

• Intermittent SSPI that tracks the source machine or the DC, not the target, is an encryption-type problem — not an SPN problem. Stop editing SPNs.
• klist’s KerbTicket Encryption Type and Kdc Called are the two lines that matter.
• Explicitly set msDS-SupportedEncryptionTypes on your service accounts. Relying on the null default is what turned a dormant issue into a landmine.
• Keep your DCs consistent. Authentication that depends on luck isn’t authentication.

Final Word

The RC4 pin can feel like giving up — you’re deliberately stepping onto the protocol Microsoft is trying to kill. But used the way we used it here, as a no-downtime bridge to a proper AES fix, it’s exactly the right move: it stops the bleeding today and gives you room to do it right on your schedule.

If you’d rather not spend two days proving this out on your own AG, SQL Solutions Group has been untangling SQL Server high-availability and authentication gremlins for years. We’re always glad to help!

The post Intermittent “Cannot Generate SSPI Context” in SQL Server? Here’s the 2022 Kerberos Trap (and How to Fix It) appeared first on SQL Solutions Group.

They’ve been a fully supported option for SQL Server since 2014, they work with Failover Cluster Instances and Availability Groups, and Active Directory rotates the passwords for you on a schedule you control. We use them by default on every new SQL Server build at SSG.

And yet, in a decade of Health Checks, we still rarely see them deployed. The most common reasons we hear: “I tried it once and SPNs broke,” or, “I wasn’t sure it would work with our AG.” Both are addressable. Here’s what you need to know to deploy gMSAs successfully.

Why gMSAs Win

A gMSA is an Active Directory account whose password is generated by the Key Distribution Service (KDS), rotated automatically (default every 30 days), and retrievable only by computer accounts you explicitly authorize. That gives you four things:

• Passwords you don’t know, can’t leak, and can’t forget to rotate. The password is 240 bytes of cryptographic randomness. No human ever sees it.
• No service restart on rotation. When AD rotates the password, the SQL Server service keeps running. This is the part most people don’t believe until they see it.
• Automatic SPN management. The account can register and update its own Service Principal Names, eliminating one of the most painful manual chores in Kerberos troubleshooting.
• Cluster-aware. Unlike standalone Managed Service Accounts (sMSAs), a gMSA can be used by multiple computers, which means it works for Failover Cluster Instances and Availability Group replicas.

If you’re still managing SQL Server service accounts as regular AD user accounts with a 90-day password expiry policy and a coordinated change window, you’re doing work that AD will do for you free.

Prerequisites

Before you create your first gMSA, the domain needs a KDS root key. This is a one-time, domain-wide setup step. By default, AD enforces a 10-hour wait after key creation before the key becomes usable, which is a safety mechanism to ensure replication completes across all domain controllers.

Source code

# On a domain controller (or any machine with AD module)
# Production: just run this and wait 10 hours
Add-KdsRootKey -EffectiveImmediately
 
# Lab/test only: bypass the 10-hour wait
Add-KdsRootKey -EffectiveTime ((Get-Date).AddHours(-10))

Verify a key exists before proceeding:

Source code

Get-KdsRootKey

You also need:

• Windows Server 2012 or higher on the SQL Server hosts (we strongly recommend 2019+)
• SQL Server 2014 or higher (every supported version qualifies)
• The RSAT Active Directory PowerShell module on whichever machine you’re using to create the accounts

Account Structure: One Per Service, Per Server (Mostly)

The pattern we deploy at SSG: a separate gMSA for each SQL Server service type (Database Engine, SQL Agent, SSRS, SSAS) on each standalone server. For clustered services — FCIs and AG replicas — we use a single gMSA shared across all nodes in that cluster.

Why separate accounts per service type? Least privilege. Your SQL Agent account often needs different rights than your Database Engine account (proxy operations, file system access for job output, etc.). Putting them on the same account means you grant the union of all permissions to both, which is exactly what we’re trying to avoid.

Why a single account across cluster nodes? Because all nodes need to authenticate as the same identity to the network. The whole point of an FCI or AG is that clients connect to a virtual name that can move; the underlying service identity must be consistent.

A naming convention that holds up over time:

Heads up: gMSA sAMAccountName is capped at 15 characters (with a $ suffix that AD adds automatically). Long server names will collide with this. Plan your naming convention before you start creating accounts. That is why the suggested naming convention above does not have a lot of extra fluff like gmsa_svc_sql_<server> or the like. Every gmsa account will have a $ at the end so that’s how it is clear that it is a service account and is a gmsa.

Creating the Account

Two-step process: create an AD security group containing the computer accounts that will use the gMSA, then create the gMSA itself referencing that group. The security group is the indirection layer that makes gMSAs cluster-friendly — adding a new node to an FCI or AG later means adding its computer account to the group, not creating a new gMSA.

Source code

# 1. Create the security group, add computer accounts
New-ADGroup -Name 'gsg_SQL_PRODAG01' \
    -GroupScope Global -GroupCategory Security \
    -Path 'OU=Service Groups,DC=contoso,DC=com'
 
Add-ADGroupMember -Identity 'gsg_SQL_PRODAG01' \
    -Members 'SQLNODE01$','SQLNODE02$','SQLNODE03$'
 
# 2. Create the gMSA
New-ADServiceAccount -Name 'SQLEng_PRODAG01' \
    -DNSHostName 'PRODAG01.contoso.com' \
    -PrincipalsAllowedToRetrieveManagedPassword 'gsg_SQL_PRODAG01' \
    -ManagedPasswordIntervalInDays 30 \
    -ServicePrincipalNames @(
        'MSSQLSvc/PRODAG01.contoso.com',
        'MSSQLSvc/PRODAG01.contoso.com:1433'
    ) \
    -Path 'OU=Service Accounts,DC=contoso,DC=com'
 
# 3. Reboot each node OR run gpupdate /force + klist purge -li 0x3e7
#    so the computer picks up its new group membership

That last step is the one most people skip and then spend an hour debugging. Computer Kerberos tickets are issued at boot. Adding a computer to a new security group requires a reboot or a TGT refresh before the computer can retrieve the gMSA password.

Then on each SQL Server host:

Source code

# Install the AD PowerShell module if not present
Install-WindowsFeature RSAT-AD-PowerShell
 
# Install and verify the gMSA on this host
Install-ADServiceAccount -Identity 'SQLEng_PRODAG01'
Test-ADServiceAccount -Identity 'SQLEng_PRODAG01'   # must return True

Test-ADServiceAccount returning True is the gate. If it returns False, do not proceed to the SQL Server side — the problem is upstream in AD or group membership.

Assigning the gMSA to SQL Server

Use SQL Server Configuration Manager. Always. Not services.msc, not Server Manager, not the Services snap-in. SQL Server Configuration Manager grants the necessary local rights (Log on as a service, Lock pages in memory, file system ACLs, registry ACLs) automatically. The other tools don’t, and you’ll spend the rest of your afternoon figuring out why SQL Server won’t start.

In Configuration Manager: right-click the SQL Server service, Properties, Log On tab, enter CONTOSO\SQLEng_PRODAG01$ in the account name field. Two things to remember:

• Always include the trailing $. This is how Windows distinguishes a managed service account from a regular user account.
• Leave the password fields blank. If you type anything in them, the dialog will reject the save.

Repeat for SQL Agent, SSRS, etc., each with its own gMSA. Then restart the services.

Availability Groups and FCIs: The Extra Step

For Always On Availability Groups, after switching the Database Engine service to a gMSA, you need to grant the new account CONNECT permission on the HADR endpoint on every replica. This is the step that breaks AGs silently — the cluster looks fine, but synchronization stops.

Source code

USE master;
 
GO
 
CREATE LOGIN [CONTOSO\svc_SQL_PRODAG01$] FROM WINDOWS;
 
GRANT CONNECT ON ENDPOINT::Hadr_endpoint TO [CONTOSO\svc_SQL_PRODAG01$];
 
GO

Run that on every replica. If you forget one, the missed replica will fall out of sync the moment its old service account credential expires.

For FCIs, no equivalent step is needed — the gMSA is just the service account, and the cluster resource picks it up.

Deploying at Scale with dbatools

If you’re rolling gMSAs across an estate, do it with dbatools. The Update-DbaServiceAccount command handles the local rights assignment correctly — same as Configuration Manager, but scriptable across many servers in a single pipeline.

Source code

# Switch the Database Engine on a list of servers to a gMSA
$instances = 'SQLNODE01','SQLNODE02','SQLNODE03'
 
Get-DbaService -ComputerName $instances -Type Engine |
    Update-DbaServiceAccount -Username 'CONTOSO\SQLEngPRODAG01$'
 
# Verify it took
Get-DbaService -ComputerName $instances -Type Engine |
    Select-Object ComputerName, ServiceName, StartName, State

Note the lack of a -Password parameter. dbatools detects the trailing $ and treats the account as a managed service account, skipping the password prompt entirely. This is the kind of small touch that makes dbatools the right tool for this job.

Common Pitfalls and How to Diagnose Them

“Cannot generate SSPI context”

This is the canonical Kerberos failure. After switching to a gMSA, clients can’t connect via Kerberos and either fall back to NTLM (slow, less secure) or fail outright. The cause is almost always SPN-related — either the SPNs didn’t transfer to the gMSA, or duplicates exist on the old account.

Diagnose:

Source code

# Check what authentication SQL is using
SELECT auth_scheme FROM sys.dm_exec_connections WHERE session_id = @@SPID;
-- Should return KERBEROS, not NTLM
 
# Find any SPN registered for your SQL service
setspn -Q MSSQLSvc/PRODAG01.contoso.com
setspn -Q MSSQLSvc/PRODAG01.contoso.com:1433
 
# Find duplicates (these break Kerberos entirely)
setspn -X

Fix: remove SPNs from the old service account, ensure they’re on the gMSA. If you registered SPNs in the New-ADServiceAccount command, AD will refresh them automatically going forward — that’s the gMSA SPN management benefit. If you didn’t, register them manually now and let AD take it from there.

Linked servers and cross-server delegation

If your SQL Server uses linked servers configured for Windows authentication pass-through, you need Kerberos delegation set on the gMSA. In AD Users and Computers, the gMSA must be configured for constrained delegation to the target SQL Server’s MSSQLSvc SPN. This is the same requirement as for any service account — gMSAs don’t change it — but it’s worth checking before you migrate, because constrained delegation settings don’t migrate automatically when you change accounts.

Service won’t start after the switch

Almost always one of:

• The gMSA wasn’t installed on the host (skipped Install-ADServiceAccount)
• The computer account isn’t in the security group (or wasn’t rebooted after being added), or
• The account was assigned via the wrong tool and didn’t get Log on as a service.

The error in the event log is usually generic; the diagnostic order is: confirm Test-ADServiceAccount returns True, confirm group membership, redo the assignment via Configuration Manager.

When Not to Use a gMSA

Two scenarios where we don’t recommend gMSAs:

• Servers in domains where you don’t control the KDS root key. You’ll need a Domain Admin to create that one-time. If your AD team won’t, it’s a non-starter.
• Cross-forest scenarios. gMSAs do not work across AD forest boundaries. If your SQL Server needs to authenticate to resources in a different forest, you need a regular service account or a different design.

Bottom Line

gMSAs are the right default for service accounts on any domain-joined SQL Server built in the last decade. The setup overhead is real but one-time. The ongoing operational savings — no password rotations, no expiry surprises, no “who has the spreadsheet of service account passwords” conversations — pay for it many times over.

If you’re standing up a new SQL Server in 2026 and you’re not using a gMSA, you should have a specific reason why. “We’ve always done it this way” isn’t one.

Need help rolling gMSAs out across an existing estate?

SSG has done this dozens of times across customer environments — including the messy migrations from legacy service accounts on production AGs. Get in touch with us and we’d be happy to support you.

The post Group Managed Service Accounts and SQL Server: What You Actually Need to Know appeared first on SQL Solutions Group.

If you’ve encountered a situation where none of your SQL Server Always On Availability Group (AG) replicas become PRIMARY after a cluster failure — you’re not alone.  We recently had a customer with this exact scenario (AG won’t become primary after force quorum), and it is both uncommon and difficult to troubleshoot so I thought it would be worth posting about.

• What causes this issue
• Why all replicas can get stuck in SECONDARY state
• How to resolve the issue safely

The Scenario

You have a two-node Always On AG across two subnets. Due to a network failure or maintenance in which one of both subnets become unavailable:

• The Windows Server Failover Cluster (WSFC) loses quorum
• The second node is unreachable or has been evicted
• In Failover Cluster Manager, the AG Role is in a Failed state and will not come online.
• In SQL Server Management Studio:
  • Both replicas show as SECONDARY
  • SQL Server refuses to allow any failover with the following error:

“The availability replica on this instance cannot become the primary replica because the WSFC cluster was started in force quorum.”

What’s Actually Happening?

This is SQL Server protecting you from a split-brain scenario. When you Force the Cluster Online due to quorum issues, SQL Server doesn’t trust that it’s safe to assign a PRIMARY replica.

So, it intentionally refuses to promote any replica to PRIMARY — even if the cluster seems healthy on the surface. This is referred to as split-brain, passive state mode. It is protecting itself since it doesn’t know which node the cluster is going to choose when / if it brings the AG role online. In this state, all the databases are in Not Synchronized, and you can’t even remove them from the AG since that has to be done on the PRIMARY replica.

Resolution

Step 1: Fix the Cluster

In the scenario we encountered, the root cause was a DNS issue with both the Cluster IP and the Listener IP on the secondary subnet, causing the IP Addresses to show up as Failed instead of Offline. Fixing the DNS issue and restarting the cluster resolved this.

This clears the “forced” quorum state and allows SQL Server to trust the cluster again.

Step 2: Restart SQL Server

This forces a recheck of the AG and WSFC state but will not fix the AG. Multiple reboots, cluster restarts, service restarts will not do that. Remember, it is protecting itself.

Step 3: Force Failover Allow Data Loss

This is a special command that should only be used in cases like this. Although allow data loss sounds scary, in this case it is quite safe since both replicas are secondaries, and no data has changed.

The command should be run on the secondary that you want to be primary, ideally the same server that was primary prior to the cluster failure.

ALTER AVAILABILITY GROUP [YourAG] FORCE_FAILOVER_ALLOW_DATA_LOSS;

This command promotes the current replica to PRIMARY even if it wasn’t fully synchronized. It basically tells SQL Server to skip the checks with the WSFC and to trust you that this should be the primary.

🔍 Post-Failover Cleanup

After promotion, check the state of your AG:

SELECT 
  replica_server_name,
  role_desc,
  synchronization_state_desc,
  connected_state_desc
FROM sys.dm_hadr_availability_replica_states;

You will likely need to resume data movement on the databases to get the secondary back in sync and, in worst case scenarios, you may have to reseed the databases on the secondary

Microsoft Docs Reference

For more details, see Microsoft’s official documentation: Perform a Forced Manual Failover of an Availability Group (Microsoft Docs)

Lessons Learned

• SQL Server is cautious with failovers — and that’s a good thing
• Forced quorum is a tool, not a fix — clear it as soon as possible
• AGs won’t elect a PRIMARY without trust in WSFC’s integrity
• Document and test your disaster recovery plan in lower environments

Final Word

The `FORCE_FAILOVER_ALLOW_DATA_LOSS` command can feel scary, but when used correctly — as in this scenario where both nodes thought they were SECONDARY — it’s the right call.

Knowing how to handle these edge cases will keep your high availability setup truly available — even in the worst-case scenarios.

The post SQL Server AG Won’t Become Primary After Force Quorum? Here’s Why and How to Fix It appeared first on SQL Solutions Group.

In this post, I’ll walk you through a real-world situation tied to SQL Server object ownership where a timebomb went off. The two best practices involved here are:

• Don’t put user objects in system databases.
• Ensure the default built-in logins sa or dbo. own all objects, databases, jobs, etc.

Patching Time Bomb

Recently while installing a SQL Server service pack on one of our customer’s servers (fortunately a non-production instance), the SQL Server service would not start after reboot. I checked the usual suspects (permissions, service account issues) and checked if the Windows Event Log could give me a clue. There was nothing except the generic error in the System log that the service could not start.

So it was time to dig into the SQL Server Error Log. Even though the service would not start, the log is just a text file and does not need to be accessed through the server. The default location of the log is C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\Log, replacing the version and instance folders as needed.

Note that there is a file for each error log as they are cycled and retained. So the log from the most recent restart (or restart attempt, in this case) is named ERRORLOG, with the archived logs named ERRORLOG.# for each of the archived logs.

Looking at the log in this particular case, I noted the following errors.

Anytime you see something that says “restore the master database from backup” it’s not a good thing. However, looking a bit further back we see that the initial error indicates there is already an object named DatabaseMailUserRole in the database. That is followed by a CREATE SCHEMA failure, all of which took place during the msdb upgrade script. So the next step is to dig into the msdb database and find out what’s going on.

But if the service won’t start, how do I get into the database and figure out what’s going on with those objects?

Enter Trace Flag 902

We use Trace Flag 902 for this exact purpose. When an upgrade goes wrong, every time you start the service it will attempt to run the upgrade script. Doing so generates the same failure. This is because the instance is in “Script Upgrade Mode.” Trace Flag 902 tells SQL Server to startup and recover databases without running any pending upgrade scripts. That way you can get in, find and fix the problem, then restart the server again without the trace flag.

The easiest way to start the service with a temporary trace flag is from the command line.

After executing that command, the service started and all of the databases came up normally.

Looking into the actual issue, I found that the script deleted DatabaseMailUserRole as it was not there. Digging a little deeper, it turns out there was a second database role owned by that role, which in turned owned the schema in question. So when the upgrade script tried to upgrade that schema, it failed because it was effectively orphaned. Modifying the ownership of the schema to dbo (as it should be) and restarting the instance without the trace flag fixed the problem and the upgrade completed. I ran the Service Pack installer a second time to be sure it got everything. Sure enough it said “Incomplete” for the database engine. This time the install completed and rebooted with no issues.

Conclusions

So we see an issue with two Best Practices here:

1. Creation of user objects (in this case a role) in the msdb database.
2. Changing the SQL Server object ownership of a schema to use that role.

I don’t know how long it had been this way nor how it got that way. But it’s a great example of something that is just sitting there, seemingly benign, until the ticking stops and the bomb goes off.

The post Ticking Timebombs: Object Ownership and System Databases Gone Wrong appeared first on SQL Solutions Group.

SQL Server 2000 introduced UDFs. While they can be useful in certain situations, an all too prevalent anti-pattern has emerged.

SQL User Defined Functions: Identifying The Problem

To illustrate this, let’s take a simple example from the AdventureWorks  sample database. If you don’t have this database, you can download and install it. This is a good thing to have on your personal development or lab SQL Server instance, as many blogs, books, and articles use it for examples.

In our example, we are going to generate a simple report for the AdventureWorks Human Resources department using the following tables.

Our report will consist of the employee’s name, the department they work in, their job title, and start date. We have a mock-up of the report that the end user would like to see:

Taking a look at the schema above, we realize that the EmployeeDepartmentHistory table has most of this information. So we start there.

SELECT BusinessEntityID AS EmployeeID,
       DepartmentID,
       StartDate
FROM HumanResources.EmployeeDepartmentHistory

This gets us the EmployeeID and DepartmentID, but not the actual names. As developers, we are big into encapsulating code as much as possible to avoid repetitive code and the manageability issues that come with it. We know that UDFs in SQL Server can help with this so we create the following function to get the department name:

CREATE FUNCTION dbo.GetDepartment
(
    @DepartmentID INT
)
RETURNS VARCHAR(40)
AS
BEGIN
    DECLARE @ret VARCHAR(40)
    SET @ret =
    (
        SELECT [Name]
        FROM HumanResources.Department
        WHERE DepartmentID = @DepartmentID
    )
    RETURN @ret
END

Now we can execute the same query but call the new function to return the department name.

SELECT
     BusinessEntityID AS EmployeeID,
     dbo. GetDepartment(DepartmentID) AS DepartmentName,
     StartDate
 FROM HumanResources. EmployeeDepartmentHistory

That worked great so we do the same thing for EmployeeName and JobTitle.

CREATE FUNCTION dbo. GetEmployeeName(@EmployeeID int)
RETURNS VARCHAR(40)
AS
BEGIN
  DECLARE @ret VARCHAR(40), @ContactID INT 

  --get contactid
  SELECT @ContactID = BusinessEntityID FROM HumanResources. Employee WHERE BusinessEntityID = @EmployeeID

  --get name
  SET @ret = (	SELECT coalesce(FirstName,'') + ' ' + coalesce(MiddleName,'') + ' ' + coalesce(LastName,'')
				FROM Person. Person
				WHERE BusinessEntityID = @ContactID
			  )

  RETURN @ret
END

CREATE FUNCTION dbo.GetJobTitle
(
    @EmployeeID INT
)
RETURNS VARCHAR(40)
AS
BEGIN
    DECLARE @ret VARCHAR(40)
    SET @ret =
    (
        SELECT [JobTitle]
        FROM HumanResources.Employee
        WHERE BusinessEntityID = @EmployeeID
    )
    RETURN @ret
END

Executing the query again using the three functions we’ve created, we get the data we are looking for. We have encapsulated the code into function for re-use in other places, and we avoided those pesky joins. The resulting query is clean and simple. Best of all, this is a single query so we’re doing a set operation like our DBA has told us we should do.

SELECT
	dbo. GetEmployeeName(BusinessEntityID) as EmployeeName,
	dbo. GetDepartment(DepartmentID) AS DepartmentName,
	dbo. GetJobTitle(BusinessEntityID),
	StartDate
FROM HumanResources. EmployeeDepartmentHistory

So what’s the problem?

The problem is that we have unwittingly turned our single query into a loop by introducing these functions. Because the functions are non-deterministic, they have to be called for each row in our result set.

To see this, let’s fire up our old friend SQL Server Profiler, which shows us the actual executing statements and the accompanying resources used. We’ll use the following trace definition to keep things as simple and clean as possible:

Let’s execute our query and see what we get.

SELECT
	dbo. GetEmployeeName(BusinessEntityID) as EmployeeName,
	dbo. GetDepartment(DepartmentID) AS DepartmentName,
	dbo. GetJobTitle(BusinessEntityID),
	StartDate
FROM HumanResources. EmployeeDepartmentHistory

Wait a minute. Our single-statement query just generated 2076 individual SQL Statements, generating a total of 2,672 logical reads. If we look through the profiler results carefully, we can see that each of the three functions was called once for each of the 296 rows our query returned. So what looks like a simple set operation is, in fact, a loop.

We go to our DBA for help and she explains that we need to get rid of the functions and use joins instead to make it into a true set operation. So we refactor the query as follows:

SELECT
	COALESCE(c. FirstName,'') + ' ' + coalesce(c. MiddleName,'') + ' ' + coalesce(c. LastName,'') AS EmployeeName,
	d. name AS DepartmentName,
	e. JobTitle,
	StartDate
FROM HumanResources. EmployeeDepartmentHistory eh
	INNER JOIN HumanResources. Department d ON eh. DepartmentID = d. DepartmentID
	INNER JOIN HumanResources. Employee e ON e. BusinessEntityID = eh. BusinessEntityID
	INNER JOIN Person. Person c ON c. BusinessEntityID = e. BusinessEntityID

Now if we execute the query and look at the profiler trace, we get just a single statement for a total of 1032 logical reads. Our query has become much more complicated and we don’t have the benefit of code encapsulation, but we’ve got the logical reads to less than half. And that was for only 296 rows.

Conclusion

This is a simple example, but hopefully you can see the impact in a busy system with lots more data. I have seen this in production environments where something like GetCustomerName() is being called tens of millions of times daily.  This is a great example of an anti-pattern, where SQL user defined functions look like a great idea but the performance impact can be devastating.

The post T-SQL Anti-Patterns: SQL User Defined Functions (UDFs) that turn your set operation into a cursor appeared first on SQL Solutions Group.

A great example of this is going back to when SQL 2005 was released. This was a massive change to SQL Server with the introduction of SQL Server Management Studio (SSMS), Dynamic Management Objects (i.e. DMVs and DMFs) and much more. For those of us who had been using the product for quite some time, getting rid of Enterprise Manager and Query Analyzer was big deal and a steep learning curve.

DMVs were even worse. To this day I still find myself querying old system tables such as sysfiles or sysprocesses when I’m not thinking about it. In fact, in the spirit of full disclosure, I did not really focus on using DMVs and upgrading those skills until 2008 or so when I attended a pre-conference workshop on DMVs given by Kevin Kline (Twitter).

What does this have to do with being a DBA in 2021? Everything. We have seen massive changes to the Microsoft Data Platform over the last 5-10 years — PowerShell, Azure, other cloud platforms, and the advent of technologies such as NoSQL databases and Big Data to name a few.

In 2018, Microsoft released Azure Data Studio (ADS). ADS is a fork of Visual Studio Code, so if you’ve used that at all it will be very familiar. Like many others, I have not been super fast to adopt and start using it.

In the spirit of skilling up, my co-worker Scott Klein and I were discussing this a couple of weeks ago and we hatched a plan: For the next 30 days, we are both going to force ourselves to use ADS for everything we do.

Then, on March 17th, Scott will host a Webinar on SSMS vs. ADS and what our experience has been. Watch for a blog from Scott in the next day or two for background on this battle of the apps.

Fasten your seat belts! This is going to be a blast.

The post Skill Up or Give Up: Azure Data Studio or Bust appeared first on SQL Solutions Group.

Power Plan

Disk Layout

NTFS Block Size

SQL Server Service Accounts

Windows User Rights

• Lock Pages in Memory (LPIM) – This allows SQLOS to better manage memory and avoid paging to disk.
• Perform Volume Maintenance Tasks (PVMT) – This enables Instant File Initialization (IFI) so that you don’t block data file creation and growth operations while the new space is zeroed.

Summary

The post Five Tips: Configure Windows for Better SQL Server Performance appeared first on SQL Solutions Group.

In all versions of SQL Server between 2005 and 2012, the IDE for Business Intelligence projects has been included as part of the SQL Server installation. Based on Visual Studio, it gives you the ability to create BI projects in the Visual Studio Environment. In SQL Server versions 2005 through 2008 R2, it was called Business Intelligence Development Studio (BIDS). In SQL Server 2012 it was renamed to SQL Server Data Tools (SSDT) but was basically the same thing.

For some inexplicable reason, in SQL Server 2014 Microsoft decided not to include SSDT as part of the normal SQL Server installation. It is now a separate download. What is even more confusing is that there are different downloads depending on what you already have. If you go to the SQL Server Data Tools download page on MSDN, there are several options for you to choose from.

Need to upgrade the SQL Server skills of your team?

Leverage our expertise to create your own internal experts.

Learn More

If you have Visual Studio 2013

SQL Server tooling in Visual Studio 2013 – all the great database tools, now acquisition and updates are fully integrated in Express for Web, Express for Windows Desktop, Professional, Premium, and Ultimate. Since SQL Server tooling is included in VS, the updates will be pushed through VS Update and users will be prompted when VS is open. If you’d like to check for updates manually, open Visual Studio 2013 and choose the Tools > Extensions and Updates menu. SQL Server tooling updates will appear in the Updates list.

Download Visual Studio 2013 with SQL Server Tooling

What is confusing is that from the download page the link sends you to, there is no mention of SSDT. You can download the various versions of Visual Studio Express or a 90-day trial of the full version of Visual Studio, but this still doesn’t get you SSDT. You have to add SQL Server tooling via VS Update. So if you don’t already have Visual Studio 2013, keep reading.

If you have Visual Studio 2012

SSDT Visual Studio 2012 – provides a stand alone install experience as well as full integration into the Visual Studio Professional, Premium, and Ultimate SKUs. We are publishing the release through this page and the update feed.

Download SSDT for Visual Studio 2012

This link leads you to a download which allows you to add SSDT to Visual Studio 2012.

If you don’t have Visual Studio at all or have an older version

SSDT-BI, SQL Server Business Intelligence is a distinct toolset from SSDT or the SQL Server database tooling in Visual Studio 2013. As part of the SQL Server 2014 release, SSDT-BI has released a version for Visual Studio 2013. For details, visit the AS team blog. For support of SSDT-BI, please post to their blog or AS forums.

Download SSDT-BI for Visual Studio 2012

Download SSDT-BI for Visual Studio 2013

This is the option that most SQL Server DBAs and developers are going to need to download and install. If you don’t have a licensed copy of Visual Studio 2012 or 2013 this is what you need.

Still Confused?

Just download SSDT-BI and be done with it. If you already have Visual Studio 2013 but don’t have the BI projects, they’ll be added. If you don’t have Visual Studio 2013, you’ll get the SSDT version of it with just the BI projects.

The post What happened to SQL Server Data Tools in SQL Server 2014? appeared first on SQL Solutions Group.

At SSG, this is nothing new to us, and many IT professionals are well acquainted with working remotely. We’ve been a virtual company for our nearly 10-year history. Currently, our team is literally coast-to-coast: Seattle, L.A., Las Vegas (or even Mexico, making us international), and Orlando.

We’re sharing five best practices tips from our deep experience to make this new adventure work for you. 

1. Make sure you’re working from home, and not living at work. Out of a sense of duty or obligation, you may be tempted to work more than usual, to drag out your workday longer than you really need to. You might think, “It’s 8 p.m., but I can just take a minute to answer this email,” or “I can bang out this task in a second.” While well-intentioned, it can lead to burn out. Avoid “living at work” by setting boundaries — boundaries that protect your employer’s time as well as your personal time.
2. Establish a routine to make your work day as normal as possible. A consistent routine will help you with those boundaries. Get up at the normal time, start your day the way you usually do (as far as possible…going to the gym is likely out of the question), and be at your desk (as it were) at the regular time. I love the idea of getting dressed for work, and when work is over, go casual like you normally would. Take a lunch break, go for a walk, do something to get away from your desk for a few minutes every hour. After all, at work you’re not chained to your desk, so don’t behave that way while working at home. 
3. Claim and define your workspace. Working from home likely means you won’t be alone. For psychological as well as practical reasons, it’s important to have a defined area for work. If you don’t have a home office, you may have to improvise. High traffic areas like the kitchen and living room are poor options, so you may have to stick to your bedroom. Wherever it is, let your family know what when you’re “at work,” you’re at work, and disturbances should be kept to a minimum. A closed door should also signify “do not disturb” when needed. 
4. Use the tools that facilitate remote work. As IT professionals, we are (or should be) well versed in the tools that make working from home as seamless as possible. Tools like Asana, Jira, Zoom, Slack and Microsoft Teams are indispensable to keeping the collaboration, communication, and project management moving forward. Hopefully you and your organization are already established with one or more of these (or similar) and you don’t need a crash course in how to use them. If not, maybe you need to establish yourself as a SME and get the ball rolling. And, bonus tip: Before you have a Zoom call or something similar, build in some time to ensure you’ve addressed any installation needs and that your camera, mic, and speakers are good to go. (By the way: did you see where the CEO of Zoom is removing video conferencing time limits for K-12 schools in Japan, Italy and the U.S.? Well done, Eric Yuan.)
5. Take a deep breath. If working from home is new to you, give yourself time to adjust. Many others are in the same boat and will give you the benefit of the doubt. If you’re on a conference call and the dog barks (it will happen), don’t freak out. Use good etiquette and common sense in your on-line interactions, but remember that real life will continue to happen. Here’s a good way to cope with the stresses of our new reality: If you usually have a lengthy commute, fill up that time with some reading, exercise, catnap or something else that helps you clear your head. 

I’ve seen some commentators suggest that the necessity of employees working from home now will lead to many companies increasingly  transition to such an arrangement following the pandemic. I won’t go that far, but it’s true that IT work in general and database management in particular is highly compatible with working remote. 

Regardless of how long this lasts for you, making it feel as normal as possible will be a key to how successful your time away from the office goes. I’d love to hear how you’re making it work for you, so leave your comments. 

The post Social distancing in IT: Five tips to make working from home work for you appeared first on SQL Solutions Group.