[{"categories":["Tools"],"contents":"I trusted AI to redesign this entire blog. The layout, the CSS, the deployment pipeline, the shortcodes. I use it to build home automation, organize my notes, fix my English. But I won't let it write a SQL query against a production database.\nThat's not a contradiction. That's risk management.\nThe irony AI is one of the reasons I stopped blogging. The scraping, the attribution anxiety, the feeling that anything I write gets ingested into a model and regurgitated without credit. Why spend hours testing and documenting when the output gets laundered through a chatbot?\nAnd then AI is the reason I started again. Not because I got over those concerns (I didn't). But it turns out AI is spectacularly good at all the things that aren't writing. The stuff that kept me from blogging wasn't the blogging itself. It was the everything else. The theme that needed updating. The CSS that looked dated. The deployment that was fragile.\nAI handled all of that. Not perfectly, not on the first try, but well enough that I could focus on what I actually wanted to do: start blogging again.\nI've seen this pattern before. I resisted smartphones for years. Privacy concerns, data collection, the feeling that Google knows too much about me. My wife eventually got one for me and I gradually started using it more and more. Now I'm the family tech support. The utility won. Not because the concerns went away, but because I got tired of the inconvenience of fighting it.\nThe same thing happened with AI. I'm not sure that's a good lesson. It might just be a true one.\nHow I got here I didn't wake up one day and hand my entire workflow to AI. It was a progression, and each step required the previous one to not blow up in my face.\nStage 1: The chatbox I started chatting. Mostly around unfamiliar topics, but also testing how well it knows the things I know a lot about. Two things hooked me early.\nFirst, grammar. I'm an ESLEnglish as a Second Language\nA non-native English speaker. speaker and AI turned out to be a patient, non-judgmental editor. Something that understands what I'm trying to say and helps me say it better.\nSecond, research. Not the \"give me an answer\" kind but the interactive kind. I'd go in with an open mindset, knowing I don't know enough to ask the right question yet. Ask broadly, follow threads, let the conversation narrow down. It's the opposite of the XY problem - instead of asking how to implement my premature solution, I'd describe the goal and let the tool help me find the right approach.\nTip Learn how your AI tool works before you try to get value from it. It's like reading the manual on a new appliance. You can just start pressing buttons, but you won't get 80% of the features. Stage 2: Coding outside my lane Once I understood where AI gets things right and where it falls apart, I got more comfortable letting it write code. But only in areas where I'm not the expert.\nHome automation was the big one. Sifting through container logs, debugging Docker networking, writing automation scripts. I know what I want the system to do. AI helps me get there faster, especially when I'm troubleshooting something unfamiliar at 22:00 and don't feel like reading three forums.\nCritical One area you shouldn't underestimate with home automation (or any IoT): security and disaster recovery. AI can write the config, but you need to understand the attack surface and have a recovery plan. This applies to any domain where AI helps you move fast in unfamiliar territory. Same pattern with this blog. Hugo templates, CSS layouts, PowerShell scripts. I know what the result should look like. I don't need to be an expert in Go templating to get there.\nStage 3: File access and credentials This was the leap. Going from \"chat window where I paste things\" to \"agent that reads and writes files on my machine\" felt risky. Giving it API credentials felt even riskier.\nBut it paid off. Right now I use Claude Code (but I'm not married to it). It manages my Cloudflare configuration, drafts LinkedIn posts, pulls analytics, and orchestrates the blog deployment. The key was compartmentalization: a separate credential vault with only the access it needs. Minimal permissions, read-only where possible, no production database credentials.\nWhat I actually use it for Every example here is something I did this week:\nFixing my English. Editing that understands context, not just grammar rules Research and decision-making. Comparing tools, summarizing options, interactive Q\u0026A until I understand the options Coding outside my expertise. Home automation, Hugo templates, CSS, PowerShell Reorganizing and categorizing. I dump a pile of unstructured notes and AI finds patterns, suggests categories, creates tags Admin tasks. Timesheets, status updates, ticket descriptions Diagnostics. Sifting through log files, finding patterns in noise This blog. The redesign, the content pipeline - and this very post (more on that below) Warning What all of these have in common: if AI gets it wrong, I notice fast and the cost of fixing it is low. A broken CSS rule? I see it in the browser. A bad config? The container doesn't start. The risk/benefit framework I didn't plan to have a framework. It just emerged from watching myself decide which tasks to hand off and which to keep.\nThere's an xkcd comic about whether automating a task is worth the time. How often do you do it, multiplied by how long it takes, compared against how long the automation takes to build. For most tasks, the break-even point is depressingly far away.\nAI changes that math. What used to take a weekend of scripting takes an afternoon of prompting. More tasks cross the \"worth automating\" line.\nBut the other half of the equation is risk.\nA typo in a status update? Nobody notices. A wrong UPDATE statement without a WHERE clause? No, thank you.\nSo the real question isn't \"can AI do this?\" It's: \"If AI gets this wrong, how bad is it and how quickly will I know?\"\nTip Before you hand a task to AI, answer two questions. How will you verify the output? And what happens if you don't catch a mistake? If the answers are \"easily\" and \"not much\" - go for it. Why SQL is the hard part The obvious answer is elevated access. I have sysadmin on production servers. If AI acts through my credentials, it has the keys to everything.\nBut the deeper problem is infrastructure. Despite years of progress in database DevOps, fully automated CI/CD with environments in good shape is still a rare sight in practice.\nTesting stateless code is just easier. Reset the environment, run the suite, done. Databases carry state that took months to build. Rollback means \"restore from backup and hope.\"\nSo when I say I don't trust AI with SQL, it's not just about the permissions. It's about the absence of a safety net. That's a solvable problem, and one our industry needs to solve regardless of AI.\nHow to start The advice I'd give myself two years ago:\nLearn the tool first. Understand context management, prompting patterns, and limitations. You need this foundation before the tool is useful Start where you ARE the expert. You'll catch the mistakes. If AI writes bad SQL and you know SQL, you'll spot it instantly. If it writes bad code in a domain you don't know, you won't know it's wrong until something breaks Estimate the impact. How bad is it if AI gets this wrong? How quickly can you recover? Low impact, fast recovery - go for it. High impact, slow recovery - keep doing it yourself for now Invest in the boring stuff. The biggest wins aren't flashy. They're the admin tasks, the formatting, the categorizing About this post This post was written with AI assistance, and I expect every future post will be too. The prose is just better than what I'd produce on my own.\nEvery idea and opinion is mine. I read the final version and I'm responsible for it. That's the human-in-the-loop part - not approving a draft, but owning the output.\nThe bottom line Fire is a good servant but a bad master. Same with AI. The same risk assessment that makes you a good DBA should guide how you adopt these tools.\nThe database world will get there. Better test environments, proper CI/CD, automated validation. When the safety net exists, I'll be happy to let AI help with SQL too.\nI haven't tested it yet. Not really. But I'm building toward it.\n","permalink":"/posts/my-toolbox-ai/","tags":["Productivity"],"title":"My Toolbox - AI"},{"categories":["Tools"],"contents":"What does SQL Server actually see when it parses your code? Not the query plan, not the execution - the step before that. The raw parse tree. I built a web tool to make that visible.\nWhat is an abstract syntax tree? Before SQL Server can optimize or execute anything, it has to parse the text into a structured tree. Microsoft ships a .NET library called ScriptDOM that does exactly this: it takes a string of TSQL and returns an abstract syntax tree (AST) where every keyword, identifier, expression, and clause is a typed node.\nIf you've ever written or read anything built on top of ScriptDOM - a linter, a formatter, a migration analyzer - you've dealt with this tree. The problem is that it's invisible. You write SELECT * FROM dbo.Orders and the parser quietly turns it into a SelectStatement containing a QuerySpecification containing a SchemaObjectName with dbo in the schema slot and Orders in the base identifier slot. None of that is obvious from the flat text.\nI've used the ScriptDOM library before, for instance to build a homegrown linter. You can also use it to build your own formatter with the correct set of rules.\nSurprising parses The best way to understand the AST is to throw weird SQL at it and see what happens. Here are a few that might surprise you.\nSpaces where they shouldn't be Brent Ozar showed this one in his Stupid T-SQL Tricks post. This is valid TSQL:\nSELECT * FROM sys . databases Spaces around the dot? Sure, why not. The parser doesn't care about whitespace between the parts of a multi-part name. It still resolves sys as the schema and databases as the object, exactly the same tree as sys.databases. Paste both versions into the Visualizer and compare the trees - they're identical.\nReserved words as identifiers This is also valid:\nSELECT [SELECT] FROM [FROM] WHERE [WHERE] = 1 Square brackets let you use anything as an identifier. The AST shows these as regular ColumnReferenceExpression and SchemaObjectName nodes, no different from normal column and table names. The parser doesn't know or care that the identifier text happens to be a keyword.\nMulti-part name slots How many parts can an object name have? Try this:\nSELECT * FROM ServerName.DatabaseName.SchemaName.TableName The SchemaObjectName node has four slots: ServerIdentifier, DatabaseIdentifier, SchemaIdentifier, and BaseIdentifier. When you write just FROM Orders, three of those slots are empty. When you write FROM dbo.Orders, two are empty. The Visualizer makes it obvious which slot each part fills.\nTry it out Head over to TSQL Visualizer and paste in some TSQL. The tool has three panels - Editor, Tree, and Fragment - with a three-way sync: click a token in any panel and it highlights the corresponding position in the other two.\nTry pasting your own queries and clicking around. Some things worth exploring: how CASE expressions nest, what a subquery looks like in the tree, or how a JOIN clause is structured compared to a comma-separated FROM.\nAbout the project Full transparency: this tool is a vibe-coded, AI-assisted project. I'm learning C# and Blazor, and building something useful seemed like a better way to learn than following tutorials. The code is probably not production-grade, but it works and hopefully helps others understand ScriptDOM.\n","permalink":"/posts/tsql-visualizer/","tags":["ScriptDOM","Parsing"],"title":"Inspecting TSQL Abstract Syntax Trees"},{"categories":["How to"],"contents":"Twenty stolen cars, twenty swapped plates. The obvious move - join on VIN - is exactly what the thieves counted on us trying. The cars' last known locations before the swap tell a different story.\nThe problem This is the abridged case description. Emphasis mine:\nThere has been a sudden increase in unsolved cases of stolen cars all across our city, and the police need our help again to crack the case.\nWe've been given access to a massive dataset of car traffic for over a week, as well as a set of cars that have been stolen. It's possible that the car's identification plates were replaced during the robbery, which makes this case even more challenging.\nWe need you to put on your detective hat and analyze the data to find any patterns or clues that could lead us to the location of these stolen cars. It is very likely that all the stolen cars are being stored in the same location.\nAnd the main question:\nWhere are the stolen cars being kept?\nMore specifically: Avenue and Street numbers.\nThe investigation We have two tables - StolenCars and CarsTraffic. StolenCars is just a list of 20 VINs, so let's take a look at CarsTraffic.\nCarsTraffic | take 50 The traffic data covers June 10-16, 2023 - just under a week.\nSo it looks to be VINs and their respective location at a point in time. My first instinct was to join directly on VIN - but swapped plates mean the stolen car shows up under a completely different VIN after the robbery. The VIN is useless as a stable link across the swap. The location is not.\nIf a stolen car had its plates changed at location X, then the car that drove away from X shortly afterward is the same vehicle under a new identity. The key insight: find where each stolen VIN was last seen, then find what new VIN appeared at that exact spot within a narrow time window.\nThe solution Find the last location (Ave and Street) and time of each stolen VIN Find new possible VINs - join on the last location within a certain timeframe - let's say 10 minutes Find last location of the new possible VINs Aggregate by location and see if any of them has around 20 cars Note The 10-minute window is a judgment call. Too short and you might miss a slow plate swap; too long and you start pulling in unrelated cars that happened to pass the same junction. Ten minutes felt like a reasonable first guess for a street-level operation. let LastLocation = ( CarsTraffic | where Timestamp \u003e= datetime(2023-06-10) and Timestamp \u003c datetime(2023-06-17) | lookup kind = inner StolenCars on VIN | summarize arg_max(Timestamp, *) by VIN | project-rename VinChangeTime = Timestamp | distinct Ave, Street, VinChangeTime ); let NewPossibleVINs = ( CarsTraffic | where Timestamp \u003e= datetime(2023-06-10) and Timestamp \u003c datetime(2023-06-17) | lookup kind=inner LastLocation on Ave, Street // VIN change within 10 minutes | where Timestamp \u003e= VinChangeTime and Timestamp \u003c VinChangeTime + 10m | distinct VIN ); NewPossibleVINs | join kind = inner (CarsTraffic | where Timestamp \u003e= datetime(2023-06-10) and Timestamp \u003c datetime(2023-06-17)) on VIN | summarize arg_max(Timestamp, *) by VIN | summarize count() by Ave, Street | where count_ \u003e= 20 | top 5 by count_ desc Only three locations came back with 20+ cars, so top 5 returned all three. The one with 21 is the answer: the case said all stolen cars are likely in one place, and 21 matches 20 stolen cars almost exactly. The other two clusters - 70 and 56 cars - are just busy intersections. Way too many to be a stash.\nLet's input it as an answer and grab another batch.\n","permalink":"/posts/kda-eod-case-3/","tags":["KQL"],"title":"KDA: Echoes of Deception - Case 3"},{"categories":["How to"],"contents":"The people of Digitown are being targeted by phishermen, and they need my help to stop them in their tracks.\nThe problem The police have asked for our assistance, and we've got a massive data set to work with. We've got listings of all the calls that have been made during the week, and we need to find the source of the phishing calls.\nIt's not going to be easy, but we know you're up for the challenge! We need you to analyze the data and use your detective skills to find any patterns or clues that could lead us to the source of these calls.\nAnd the main question is:\nWhat phone number is used for placing the phishing calls?\nLet's take a look at our new table PhoneCalls\nPhoneCalls | take 50 The data spans a single week - May 21 through May 27, 2023.\nLooking at the data I can infer a couple of things:\nThere are two EventTypes - Connect and Disconnect Connect has the information about origin and destination as well as if it's hidden Disconnect has info about who ended the call The time between Connect and Disconnect timestamps will also serve as the call duration So right off the bat I know I'll have to do parsing of the dynamic Properties column as well as self-join on the CallConnectionId.\nModus operandi Let's also make a few assumptions about the phisher:\nThey will make a lot of calls to unique numbers They'll prefer to hide their caller ID The calls will generally be short - either they succeed or give up on the target Most of the time the call will be ended by the potential victim Let's look at how call durations are distributed. I'm bucketing them into 5-minute bins.\n// 5 minute bucket histogram PhoneCalls | where Timestamp \u003e= datetime(2023-05-21) and Timestamp \u003c datetime(2023-05-28) | where EventType == \"Connect\" | extend tostring(Properties.Origin) , tostring(Properties.Destination) , tobool(Properties.IsHidden) | project-rename ConnectTime = Timestamp | join kind=inner ( PhoneCalls | where Timestamp \u003e= datetime(2023-05-21) and Timestamp \u003c datetime(2023-05-28) | where EventType == \"Disconnect\" | extend tostring(Properties.DisconnectedBy) | project-rename DisconnectTime = Timestamp ) on CallConnectionId | extend Duration = DisconnectTime - ConnectTime | where Properties_DisconnectedBy == \"Destination\" | summarize count() by bin(Duration, 5m) | order by Duration asc | render linechart It looks like the vast majority of calls end before the ten-minute mark.\nThe histogram backs up the third assumption: most victim-disconnected calls finish inside ten minutes. Layer on hidden caller ID and the disconnect-by-destination filter, and the conditions are set.\nThe solution PhoneCalls | where Timestamp \u003e= datetime(2023-05-21) and Timestamp \u003c datetime(2023-05-28) | where EventType == \"Connect\" | extend tostring(Properties.Origin) , tostring(Properties.Destination) , tobool(Properties.IsHidden) | project-rename ConnectTime = Timestamp | where Properties_IsHidden | join kind=inner ( PhoneCalls | where Timestamp \u003e= datetime(2023-05-21) and Timestamp \u003c datetime(2023-05-28) | where EventType == \"Disconnect\" | extend tostring(Properties.DisconnectedBy) | project-rename DisconnectTime = Timestamp ) on CallConnectionId | extend Duration = DisconnectTime - ConnectTime | where Properties_DisconnectedBy == \"Destination\" | where Duration \u003c 10m | summarize count() , count_distinct(Properties_Destination) , avg(Duration) by Properties_Origin | top 10 by count_ desc The first entry is miles ahead of the others - 186 calls to 186 distinct destinations, average under two minutes. The self-join paired origin details with duration and disconnect behaviour; the assumptions told it where to look. Another badge earned.\n","permalink":"/posts/kda-eod-case-2/","tags":["KQL"],"title":"KDA: Echoes of Deception - Case 2"},{"categories":["How to"],"contents":"Another KDA case. Digitown's utility bills suddenly doubled for no good reason. With the election coming up, I got pulled in to figure out what went wrong. I’ve got the billing data and the SQL they used - now it’s time to dig in.\nThe problem Imagine this: It's a fresh new year, and citizens of Digitown are in an uproar. Their water and electricity bills have inexplicably doubled, despite no changes in their consumption.\nSounds less like a billing error and more like Digitown accidentally subscribed to a corporate pricing model.\nThey also mention:\nThe city's billing system utilizes SQL (an interesting choice, to say the least), but fret not, for we have the exported April billing data at your disposal. Additionally, we've secured the SQL query used to calculate the overall tax. Your mission is to work your magic with this data and query, bringing us closer to the truth behind this puzzling situation.\nSELECT SUM(Consumed * Cost) AS TotalCost FROM Costs JOIN Consumption ON Costs.MeterType = Consumption.MeterType And the main question to answer is:\nWhat is the total bills amount due in April?\nSolution The SQL code is just there to highlight the EXPLAIN functionality, which translates SQL code to KQL.\nEXPLAIN SELECT SUM(Consumed * Cost) AS TotalCost FROM Costs JOIN Consumption ON Costs.MeterType = Consumption.MeterType gives\nCosts | join kind=inner (Consumption | project-rename ['Consumption.MeterType']=MeterType) on ($left.MeterType == $right.['Consumption.MeterType']) | summarize TotalCost=sum(__sql_multiply(Consumed, Cost)) | project TotalCost Tip EXPLAIN translates SQL into its KQL equivalent. Handy as a starting point when you know the SQL logic but not the KQL syntax - though the output usually benefits from cleanup. But the query is simple enough we can do without it.\nLet's start by inspecting the data. The ingestion created two tables: Consumption (actual usage) and Costs (lookup table)\nConsumption | take 50 The data covers all of April 2023:\nConsumption | summarize min(Timestamp), max(Timestamp) Note In KDA, the ingested data is already scoped to each case's time window, so a time filter won't change the results here. In production Kusto environments, always add the time filter first - the engine partitions data by ingestion time, and an early where Timestamp between(...) dramatically reduces scan cost. We'll add time filters throughout this series to build the habit. We can take a look at a random household:\nConsumption | where HouseholdId == \"DTI3F5F67DE8F2E87B8\" Looks like there is a single Water and Electricity consumption per household per day.\nLet's test that hypothesis:\nConsumption | where Timestamp \u003e= datetime(2023-04-01) and Timestamp \u003c datetime(2023-05-01) | summarize count() by HouseholdId, bin(Timestamp, 1d) | where count_ \u003e 2 | take 10 Taking a closer look at one of the overcharged households and time periods:\nConsumption | where HouseholdId == \"DTI755365C722DAEA93\" | where bin(Timestamp, 1d) == datetime(2023-04-29 00:00:00.0000) We can see some duplicate data. But wait - there’s more:\nConsumption | where Timestamp \u003e= datetime(2023-04-01) and Timestamp \u003c datetime(2023-05-01) | where Consumed \u003c 0 | take 10 This shows there are some rows where consumption is negative.\nWe need to filter out bad data and join with the Costs table to get the final result.\nNote distinct * removes exact duplicate rows - every column must match. If the duplicates differ by even one field (a slightly different timestamp, for instance), this approach won't catch them. The final query:\nConsumption | where Timestamp \u003e= datetime(2023-04-01) and Timestamp \u003c datetime(2023-05-01) | where Consumed \u003e 0 // filter out negative consumption | distinct * // remove duplicate lines | summarize sum(Consumed) by MeterType | lookup kind=inner Costs on MeterType | summarize sum(Cost * sum_Consumed) Two problems in the data: duplicate consumption records inflating every bill, and negative entries pulling the total the wrong way. Filter the negatives, dedup the rest, join with the cost table - case closed.\n","permalink":"/posts/kda-eod-case-1/","tags":["KQL"],"title":"KDA: Echoes of Deception - Case 1"},{"categories":["How to"],"contents":"The Kusto Detective Agency doesn't let you jump straight into cases. First: onboarding. Let's walk through the interface and solve the opening challenge.\nThe UI layout This is what your layout might look like:\nLHS menu where you can switch challenges. All currently available cases. You'll start with one and unlock new ones. Ordered like a mailbox, with the oldest at the bottom. Flavor text introducing the challenge. Ingestion script to load the data into your personal cluster. The main question that needs to be answered. Answer field - usually includes hints about the expected answer format. Three hints - IIRC, not all are available from the start; you might need to wait before requesting a hint. Training section - introduces concepts and simpler challenges that help you with the main one. Solving the case We need to answer this question:\nWho is the detective that earned the most money in 2022?\nWe can see that only one table (DetectiveCases) was added in the ingestion section. Let's take a look at its data.\nDetectiveCases | take 50 It looks like Bounty is a dynamic property, only populated when EventType is CaseOpened.\nWe can test that hypothesis:\nDetectiveCases | where isnotempty(Properties) | take 50 | extend toreal(Properties.Bounty) Let's also review all the rows for a single case - I'll use CASE_0521475 from the first result set.\nDetectiveCases | where CaseId == \"CASE_0521475\" Only the CaseOpened event has a bounty. I'm also assuming that only the first detective to solve a case receives it.\nThe question specifies the year 2022, and I've verified that all values fall within that range.\nDetectiveCases | summarize min(Timestamp), max(Timestamp) Tip Always add the time filter first in Kusto queries. The engine partitions data by time, so an early time range dramatically reduces what gets scanned. Final query The approach:\nFind all solved cases in 2022 and get the first detective who solved each case. Cases may have started in previous years, but only the solve date matters. Self-join the data on CaseId and parse the bounty from the Properties column. Summarize and sort bounties by detective. DetectiveCases | where Timestamp \u003e= datetime(2022,1,1) and Timestamp \u003c datetime(2023,1,1) | where EventType == \"CaseSolved\" | summarize arg_min(Timestamp, DetectiveId) by CaseId | project-rename FirstSolver = DetectiveId | lookup kind=inner ( DetectiveCases | where EventType == 'CaseOpened' | extend Bounty = toreal(Properties.Bounty) | where Bounty \u003e 0 | project CaseId, Bounty ) on CaseId | summarize sum(Bounty) by FirstSolver | top 3 by sum_Bounty desc With the onboarding case solved, let's move on to the first real case.\n","permalink":"/posts/kda-eod-onboarding/","tags":["KQL"],"title":"KDA: Echoes of Deception - Onboarding"},{"categories":["How to"],"contents":"I mostly write about SQL Server, but I have a soft spot for Kusto. The Kusto Detective Agency is a set of data-mystery challenges - I worked through them a while back and decided to replay the whole thing and write it up.\nWhat's a Kusto Detective Agency? It's essentially a series of exercises where you tackle problems and uncover insights from data. Think of it as solving a mystery, but instead of clues and suspects, you're working with data sets and queries. You can try it yourself here: Kusto Detective Agency\nWhy I am replaying the challenges There was a new challenge on June 8th, 2025 called Call of the Cyber Duty, with a grand prize of $10k. I had already completed the previous challenges, but it had been a while, and I needed a refresher to keep my skills sharp.\nI've completed both full seasons and some holiday challenges as well. The only one I skipped is Digibus Real-Time Crisis, because it involves Fabric.\nChange of order At the time of writing, the order of challenges in the menu doesn't match the chronological release of the seasons, and names have been added to them:\nSeason 1 Now named New Shadows Over Digitown 5 cases Difficulty level = master (est. completion time 20 hours) Listed second in the menu Season 2 Now named Echoes of Deception 10 cases Difficulty level = expert (est. completion time 20 hours) Listed first in the menu I'll adhere to the newly preferred order: Season 2 first, then Season 1\nBadges Season 1 (New Shadows Over Digitown) Season 2 (Echoes of Deception) Holiday Challenges ","permalink":"/posts/kusto-detective-agency-intro/","tags":["KQL"],"title":"Kusto Detective Agency - Intro"},{"categories":["How to"],"contents":"The problem I attempted to deploy a trigger change on a load-bearing table in a busy system. I was repeatedly blocked and, while waiting for the SCH-S lock, I ended up blocking other queries. I group blockers into two categories based on duration:\nLong You have enough time to detect the block and query WhoIsActive or another DMV utility to identify the blocker. Can you kill it? It might be critical for business or have a long rollback time. Short These blockers come and go so quickly you might not even notice them. You're usually blocked due to lock partitioning and your session running on a scheduler with a higher ID. (as I cover in Async stats update causing blocking) Ideally, deployments should happen when there's no active locking on the object. This reminded me of WAIT_AT_LOW_PRIORITY feature of the online index rebuilds.\nIt would be great if this option were available for all schema changes, but since it isn't, I decided to build my own version.\nBuilding blocks To build this utility, we'll need:\nA time-based loop (infinite loops can go wrong) Check for incompatible locks The actual code to be deployed Most of the code deployments are required to be in a separate batch (in SSMS the batch separator is GO by default) End gracefully if you cannot get the lock in the defined time Any other safety measures The main challenge was how to put the deployment script in a separate batch. The first method that came to mind was using dynamic SQL, but that would mean using dynamic SQL (and all its pitfalls like double apostrophes, formatting issues, lack of syntax highlighting, etc.). That's why I chose the rarely used GOTO.\nNote This gives you one GO-bordered deploy slot. If your change spans multiple batches, you will need to wrap each batch in its own copy of the loop. The script SET DEADLOCK_PRIORITY LOW SET XACT_ABORT ON SET LOCK_TIMEOUT 2000 /* milliseconds */ SET NOEXEC OFF /* resets the noexec on that's set below */ DECLARE @startTime DATETIME2(0) = SYSDATETIME() DECLARE @max_duration INT = 5 /* minutes */ /* \u003c-- INSERT THE FULL schema.objectname */ DECLARE @objectId BIGINT = (SELECT OBJECT_ID('')) /* Assuming the same DB context, but feel free to modify */ DECLARE @dbId INT = (SELECT DB_ID()) RAISERROR ('Starting the wait loop', 0, 1) WITH NOWAIT WHILE DATEADD(MINUTE, @max_duration, @startTime) \u003e SYSDATETIME() BEGIN IF NOT EXISTS ( /* Check if an incompatible lock exists */ SELECT TOP 1 1 FROM sys.dm_tran_locks AS dtl WHERE dtl.resource_database_id = @dbId AND dtl.resource_associated_entity_id = @objectId AND dtl.request_status = N'GRANT' AND dtl.request_mode = 'SCH-S' ) BEGIN GOTO gotta_go_fast END WAITFOR DELAY '00:00:00.200' /* Format: hh:mm:ss.mss */ END /* after the timeloop finishes without success, prevent execution of further code*/ RAISERROR ('Time loop finished without success', 0, 1) WITH NOWAIT SET NOEXEC ON gotta_go_fast: /* ############################################################################### Insert the deploy code in the batch block below (bordered by GO separators) ############################################################################### */ GO GO /* ############################################################################### The end of the deploy batch block ############################################################################### */ Explanation Warning Test lock-checking performance before running it in a loop. Also, ensure the actual execution plan is disabled. I start with some safety measures in a form of deadlock priority and lock timeouts Variable values are in the subquery format so you can highlight and check the values before running Inside the time loop I have the SCH-S lock check The TOP 1 1 is not necessary for the EXISTS but in case you want to test the subquery with values instead of variables If no lock is detected, jump to the gotta_go_fast: label which is right before the batch separator GO You can test around which WAITFOR delay works best for you. The goal is to limit the number of loops so it's not too resource intensive If you cannot find a window of opportunity within the @max_duration minutes we'll set NOEXEC ON to prevent any further code from running SET NOEXEC OFF is at the start of the code to reset to the original state Do not forget the trailing GO to properly terminate the batch Note The lock check and the schema change are not atomic - a session can acquire SCH-S in the microseconds between the check passing and your ALTER acquiring Sch-M. SET LOCK_TIMEOUT 2000 means the deploy fails fast rather than blocking indefinitely, but it does not eliminate the window. It reduces it. Additional help The lock partitioning and schedulers play an important role when it comes to acquiring locks as mentioned in my other blog post.\nIt helps if we run our deploy at low priority script on the highest available scheduler. That way no other process can \"jump the queue\" and block our process.\nTo check current and max scheduler we can use this snippet\nSELECT dot.scheduler_id as currentSchedulerId , dosi.scheduler_count as totalSchedulerCount , dot.session_id FROM sys.dm_os_tasks AS dot CROSS JOIN sys.dm_os_sys_info as dosi WHERE dot.session_id = @@spid If we get a low scheduler, we can just try to disconnect/reconnect to get a higher one. You can keep two sessions open and \"reroll\" the lower one until you get there.\nA more aggressive version of the script If some blocking is acceptable, replace the no-lock check with a lock threshold check and change the lock timeout to something longer. Unlike ABORT_AFTER_WAIT = BLOCKERS, neither version kills any sessions - the \"aggressive\" here means proceeding when the active lock count is low enough, not force-evicting blockers.\nSET LOCK_TIMEOUT 8000 … IF /* test for a lock count */ ( SELECT COUNT(1) FROM sys.dm_tran_locks AS dtl WHERE dtl.resource_database_id = @dbId AND dtl.resource_associated_entity_id = @objectId AND dtl.request_status = N'GRANT' AND dtl.request_mode = 'SCH-S' ) \u003c= 5 … ","permalink":"/posts/deploy-at-low-priority/","tags":["Blocking","Locking","CI/CD"],"title":"Deploy at Low Priority"},{"categories":["Opinion"],"contents":"I haven’t had a rant post in a while. There is a saying: \"Anything before the word 'but' is ignored\". I love Extended Events, but … reading the extended event file is so much pain.\nIt feels like there is a conspiracy between Microsoft and Big Pharma SQL Monitoring because the best analytics tools available in SQL Server (Extended Events and Query Store) have the worst GUI and supporting tools. I'm focusing on XE in this post.\nAn all-too-common scenario I want to check the built-in system_health or AlwaysOn_health target event files for recent events. First of all, you cannot even set time-based retention on those files so giving them 500 MB might cover anywhere between 6 months and 30 minutes of data. In the former case, you open the file via GUI and spend the next 5 minutes watching the millions of unrelated events load just so you can perhaps read the events from the last day.\nIf you think the filter can help you there, you'd be surprised. You cannot use the time filter on the date you need until that date is loaded first. If you try to filter out the unrelated events, it just starts loading from the start and filters in the background. I hope you are patient as well because if you try to scroll to the end while it's still loading, there is a good chance it will crash your SSMS (bye-bye tabs).\nDon't blink. Blink and the SSMS is dead. Don't turn your back. Don't look away. And don't blink. Good Luck. — 10th Doctor (probably) fn_xe_file_target_read_file to the rescue Surely, with GUI having these (and other) problems, the DMF is a better way to read the event data, right? Right?\nParsing the XML XE's event files are stored as .xel files which is basically an XML. While SQL Server has XML shredding capabilities, chances are the DBAs/Developers are not very good at it unless you're using a lot of Service Broker or are familiar with Extended Events already.\nBut that's ok, there are plenty of examples on the internet which you can plagiarize borrow and edit until it works. Or find the presentation from Michael Rys and learn something new.\nAt any rate, one would assume that a function for reading the xel files would return an xml data type as its result.\nYou probably see where I'm going with this.\nFunction sys.fn_xe_file_target_read_file returns event_data\tnvarchar(max)!\nSo if you want to use it with the nodes() xml method, you have to cast to xml data type first. I've also commonly seen cases where the cast to xml is evaluated multiple times per query and it's super slow.\nThat's why I always cast it as xml and insert it into a temp table to pay the conversion price only once.\nIt's not super effective, but that one is on you, fn_xe_file_target_read_file, not me.\nTime filter This was my main gripe with the GUI but this function returns a timestamp_utc\tdatetime2(7) so you can easily filter on time like this:\nSELECT ef.module_guid , ef.package_guid , ef.object_name , ef.event_data , ef.file_name , ef.file_offset , ef.timestamp_utc FROM sys.fn_xe_file_target_read_file('system_health*.xel', NULL, NULL, NULL) AS ef WHERE ef.timestamp_utc \u003e DATEADD(DAY, -1, SYSUTCDATETIME()) I bet you got 0 rows returned. That's because the function is a filthy liar: the column IS typed datetime2(7), but the predicate silently returns nothing without an explicit cast. So we must add the explicit cast to make it work.\n… WHERE CAST(timestamp_utc AS datetime2(7)) \u003e DATEADD(DAY, -1, SYSUTCDATETIME()) Which doesn't use the predicate pushdown and is instead applied as a residual.\nThere is an open feedback item Filtering output from fn_xe_file_target_read_file on timestamp_utc returns no rows which has been open since 2022 with no fix in sight. I won't be holding my breath.\nMore time filtering Maybe you're a slightly more experienced XE user, and you're using rollover files. There might be a slightly more efficient way to read your data.\nIf you want the latest data, then it's probably in the last rollover file. The function's 3rd and 4th parameters are initial_file_name and initial_offset respectively. So you can efficiently skip the unrelated files and get to the tail of the log.\nA superlative suggestion with just two minor flaws:\nOne, the only way to get the list of the files is to run the same function without the parameters. And two, the only way to get the list of the files is to run the same function without the parameters. Now, I realize that technically speaking that's only one flaw, but I thought that it was such a big one that it was worth mentioning twice.\nAlright, alright. You can use the undocumented xp_dirtree or enable xp_cmdshell to get the file names but I don't like using either in my production just for the sake of this.\nThe file format is also… interesting. Looks like this system_health_0_133733556689540000.xel. I'm not sure what the first number after the session name is, but the second one looks like ticks.\nSlightly tweaking the code from Michael J Swart 's blog Converting from DateTime to Ticks using SQL Server (because of course, dateadd cannot use bigint) we'll get this:\nDECLARE @Ticks bigint = 133733556689540000 DECLARE @DateTime datetime2 = DATEFROMPARTS(1601,1,1) SET @DateTime = DATEADD( DAY, @Ticks / 864000000000, @DateTime ) SET @DateTime = DATEADD( SECOND, ( @Ticks % 864000000000) / 10000000, @DateTime ) SELECT DATEADD( NANOSECOND, ( @Ticks % 10000000 ) * 100, @DateTime ) This gives me output 2024-10-14 05:01:08.9540000 The start year 1601 was reverse-engineered to get to the current year but it matches the start of Microsoft Epoch.\nAs I was writing this blog post Martin Smith posted an answer to my almost 2-year-old question. What are the odds? Go and read it.\nConclusion So is it all bad?\nNo, I still use it and it's the best/worst (in other words only) option for programmatically parsing the xel file from within the SQL Server. But it could be so much more…\n","permalink":"/posts/why-is-fn_xe_file_target_read_file-the-worst-sql-function/","tags":["Extended Events"],"title":"Why is fn_xe_file_target_read_file the worst SQL function?"},{"categories":["Investigation"],"contents":"I originally planned this post just as an answer to a DBA Stack Exchange question: How can I get the list of tables in all the stored procedure? After preparing a lab environment, I think it deserves its own blog post.\nThe problem The original question lists these requirements:\nInformation about all the tables referenced by a stored procedure Also, list tables from cross-DB query Return DB, Schema and Table names The currently highest-rated answer uses sys.dm_sql_referenced_entities which is a bit of an overkill for given requirements. This DMF returns column granularity so to just get the list of tables you have to aggregate.\nI think a better answer would use sys.sql_expression_dependencies (which is a subset of sys.dm_sql_referenced_entities anyway) that would give just the tables.\nBut I thought - what about views or functions or nested procedures? I'd argue those are part of the main procedure as well and they can refer to tables that are otherwise hidden.\nIt could also be turtles procedures all the way down. That means I'd have to write a recursive query or a loop. It could even have *shudders* cyclic references.\nWe'll need a lab environment to test this.\nThe lab I've used AI Claude to help me generate several objects so I can test my reference script. I haven't bothered much with formatting on this one.\nTwo databases (to test cross-db query) Scalar function, view and inline table-valued function (for nested references) The main procedure with several statements Nested procedure with cyclic reference Here it is:\nCREATE DATABASE Sideline GO CREATE DATABASE Main GO USE Sideline GO CREATE TABLE dbo.Employee ( EmployeeID int PRIMARY KEY , FirstName nvarchar(50) , LastName nvarchar(50) , Department nvarchar(50) ) GO CREATE TABLE dbo.Salary ( SalaryID int PRIMARY KEY , EmployeeID int , Amount decimal(10, 2) , EffectiveDate date ) GO CREATE VIEW dbo.EmployeeSalaryView AS SELECT e.EmployeeID, e.FirstName, e.LastName, s.Amount FROM dbo.Employee AS e JOIN dbo.Salary AS s ON e.EmployeeID = s.EmployeeID GO USE Main GO CREATE TABLE dbo.Project ( ProjectID int PRIMARY KEY , ProjectName nvarchar(100) , StartDate date , EndDate date ) GO CREATE TABLE dbo.Task ( TaskID int PRIMARY KEY , ProjectID int , TaskName nvarchar(100) , AssignedTo int , Status nvarchar(20) ) GO CREATE TABLE dbo.TimeEntry ( TimeEntryID int PRIMARY KEY , TaskID int , EmployeeID int , Hours decimal(5, 2) , EntryDate date ) GO CREATE TABLE dbo.ProjectMilestone ( MilestoneID int PRIMARY KEY , ProjectID int , MilestoneName nvarchar(100) , TargetDate date , CompletionDate date NULL , Status nvarchar(20) ) GO CREATE VIEW dbo.ProjectTaskView AS SELECT p.ProjectID, p.ProjectName, t.TaskID, t.TaskName, t.Status FROM dbo.Project AS p JOIN dbo.Task AS t ON p.ProjectID = t.ProjectID GO CREATE FUNCTION dbo.GetProjectTasks (@ProjectID int) RETURNS table AS RETURN ( SELECT Task.TaskID, Task.TaskName, Task.Status FROM dbo.Task WHERE Task.ProjectID = @ProjectID ) GO CREATE FUNCTION dbo.GetProjectStatus (@ProjectID int) RETURNS nvarchar(20) AS BEGIN DECLARE @Status nvarchar(20) SELECT @Status = CASE WHEN Project.EndDate \u003c GETDATE () THEN 'Completed' WHEN Project.StartDate \u003e GETDATE () THEN 'Not Started' ELSE 'In Progress' END FROM dbo.Project WHERE Project.ProjectID = @ProjectID RETURN @Status END GO /* Cannot create circular reference in single step, alter it after the main proc is created */ CREATE PROCEDURE dbo.NestedProc AS BEGIN SELECT te.TimeEntryID, te.TaskID, te.EmployeeID, te.Hours, te.EntryDate FROM dbo.TimeEntry AS te END GO CREATE PROCEDURE dbo.MainProc AS BEGIN CREATE TABLE #NewProjects (ProjectID int) INSERT INTO #NewProjects (ProjectID) VALUES (1) , (2) -- Statement 1: Join with temp table SELECT np.ProjectID , pm.MilestoneID, pm.ProjectID, pm.MilestoneName , pm.TargetDate, pm.CompletionDate, pm.Status FROM #NewProjects AS np JOIN dbo.ProjectMilestone AS pm ON np.ProjectID = pm.ProjectID -- Statement 2: Referencing a view, scalar and itv functions SELECT pv.ProjectID , pv.ProjectName , dbo.GetProjectStatus (pv.ProjectID) AS ProjectStatus , t.TaskName , t.Status FROM dbo.Project AS p JOIN dbo.ProjectTaskView AS pv ON pv.ProjectID = p.ProjectID CROSS APPLY dbo.GetProjectTasks (pv.ProjectID) AS t -- Statement 3: Cross-DB references (table and view) SELECT t.TaskID , t.TaskName , t.Status , esv.EmployeeID , esv.FirstName , esv.LastName , esv.Amount AS Salary FROM Main.dbo.Task AS t JOIN Sideline.dbo.EmployeeSalaryView AS esv ON t.AssignedTo = esv.EmployeeID JOIN Sideline.dbo.Salary AS s ON esv.EmployeeID = s.EmployeeID WHERE t.Status = 'In Progress' -- Statement 4: Call nested procedure with cyclic reference EXEC dbo.NestedProc END GO /* Add the cyclic reference */ CREATE OR ALTER PROCEDURE dbo.NestedProc AS BEGIN SELECT te.TimeEntryID, te.TaskID, te.EmployeeID, te.Hours, te.EntryDate FROM dbo.TimeEntry AS te -- Circular call to MainProc EXEC dbo.MainProc END GO Tests DBA Stack Exchange answer Let's run the original DBA.SE answer first. I will be checking the dbo.MainProc's dependencies for all the tests.\nSELECT DISTINCT referenced_schema_name, referenced_entity_name FROM sys.dm_sql_referenced_entities('dbo.MainProc', 'OBJECT') ORDER BY referenced_entity_name /* added for comparison with other results */ It doesn't show that the Salary table and view are from a different database but nothing that can't be fixed with an additional column.\nI'll compare the raw output with sys.sql_expression_dependencies.\nsql_expression_dependencies SELECT dsre.referenced_database_name , dsre.referenced_schema_name , dsre.referenced_entity_name , dsre.referenced_minor_name , dsre.referenced_id , dsre.referenced_minor_id FROM sys.dm_sql_referenced_entities('dbo.MainProc', 'OBJECT') AS dsre ORDER BY referenced_entity_name , dsre.referenced_minor_id SELECT sed.referenced_database_name , sed.referenced_schema_name , sed.referenced_entity_name , sed.referenced_id , sed.referenced_minor_id FROM sys.sql_expression_dependencies AS sed WHERE sed.referencing_id = OBJECT_ID('dbo.MainProc') ORDER BY sed.referenced_entity_name , sed.referenced_minor_id 1 the first result set has 23 rows due to column information and has to be aggregated to get distinct tables… 2 or filter only entries where referenced_minor_id = 0 - those refer to the object level dependency 3 the sys.sql_expression_dependencies is missing the object_id for any cross-db references I think both of these DMOs are useful and it's good to know their differences.\nThe recursive solution This is my script. It uses a recursive CTE to get all the nested references.\nI'm using a cycle detection logic that I saw Itzik Ben-Gan use somewhere.\nDROP TABLE IF EXISTS #References CREATE TABLE #References ( ObjectId int NOT NULL , DbId int NOT NULL , DbName nvarchar(128) NOT NULL , SchemaName nvarchar(128) NOT NULL , ObjectName nvarchar(128) NOT NULL , FullName AS CONCAT(SchemaName, '.', ObjectName) PERSISTED , objectType nvarchar(60) NOT NULL , NestingLevel int NOT NULL , NestingPath varchar(MAX) NOT NULL , SortPath varbinary(MAX) NOT NULL /* add indexes as needed */ ) ; WITH allReferences AS ( SELECT dsre.referenced_id AS Id , ca.DbId --, dsre.referenced_database_name --, dsre.referenced_schema_name --, dsre.referenced_entity_name , 0 AS NestingLevel , CAST ( CONCAT ( '.' , CAST(ca.DbId AS VARCHAR(MAX)) , '_' , CAST(dsre.referenced_id AS VARCHAR(MAX)) , '.' ) AS VARCHAR(max) ) AS NestingPath /* format: .dbId_ObjectId. */ , CAST(CAST(1 AS BINARY(2)) AS VARBINARY(MAX)) AS SortPath , 0 AS CycleDetection FROM sys.dm_sql_referenced_entities('dbo.MainProc', 'object') AS dsre CROSS APPLY (VALUES (ISNULL(DB_ID(dsre.referenced_database_name), DB_ID()))) AS ca(DbId) WHERE dsre.referenced_minor_id = 0 /* Removing column reference granularity */ UNION ALL SELECT dsre.referenced_id , ca.DbId --, dsre.referenced_database_name --, dsre.referenced_schema_name --, dsre.referenced_entity_name , ar.NestingLevel + 1 , CAST ( CONCAT ( ar.NestingPath /* append object to existing path */ , CAST(ca.DbId AS VARCHAR(MAX)) , '_' , CAST(dsre.referenced_id AS VARCHAR(MAX)) , '.' ) AS VARCHAR(max) ) AS NestingPath /* format: .dbId_ObjectId. */ , ar.SortPath + CAST(ROW_NUMBER() OVER ( PARTITION BY ca.dbId, dsre.referenced_id ORDER BY (SELECT @@SPID)) AS BINARY(2) ) AS SortPath , IIF ( ar.NestingPath LIKE CONCAT ( '%.' , CAST(ca.DbId AS VARCHAR(MAX)) , '_' , CAST(dsre.referenced_id AS VARCHAR(MAX)) , '.%' ) , 1 /* if nestingPath already contains the same subpath it's a cycle */ , 0 ) AS CycleDetection FROM allReferences AS ar CROSS APPLY sys.dm_sql_referenced_entities ( CONCAT(OBJECT_SCHEMA_NAME(ar.Id), '.', OBJECT_NAME(ar.Id)) , 'Object' ) AS dsre CROSS APPLY (VALUES (ISNULL(DB_ID(dsre.referenced_database_name), DB_ID()))) AS ca(DbId) WHERE ar.CycleDetection = 0 AND dsre.referenced_id IS NOT NULL AND dsre.referenced_minor_id = 0 /* Removing column reference granularity */ AND ar.DbId = ca.DbId /* object is in the same database */ ) INSERT INTO #References WITH (TABLOCKX) ( ObjectId , DbId , DbName , SchemaName , ObjectName , objectType , NestingLevel , NestingPath , SortPath ) SELECT ar.Id AS ObjectId , ar.DbId , DB_NAME(ar.DbId) AS DbName , OBJECT_SCHEMA_NAME(ar.Id, ar.DbId) AS SchemaName , OBJECT_NAME(ar.Id, ar.DbId) AS ObjectName , IIF ( o.object_id IS NOT NULL /* same DB reference */ , o.type_desc , 'ObjectInAnotherDB' ) AS objectType , ar.NestingLevel , ar.NestingPath , ar.SortPath FROM allReferences AS ar LEFT JOIN sys.objects AS o ON ar.Id = o.object_id AND ar.DbId = DB_ID() WHERE ar.CycleDetection = 0 SELECT * FROM #References AS r This uses the sys.dm_sql_referenced_entities because I don't want to lose the cross-db object_id - that only holds as long as the other database is online and its objects can be resolved. The cross-DB view dbo.EmployeeSalaryView is not expanded though because the DMOs are database-scoped. I didn't want to complicate this even further with a dynamic SQL (or linked servers for that matter).\nI'm also getting duplicates (due to nested references) and that's why I'm inserting it into temp table #References so you can slice and dice the data in any way you want it (That's the way you need it). The duplicates aren't noise - the same object appearing multiple times means it's reachable via multiple dependency paths, which can itself be useful information. For example:\nSELECT r.DbName , r.FullName , COUNT(1) AS Cnt FROM #References AS r GROUP BY r.DbName , r.FullName ORDER BY Cnt DESC This gives you each distinct object once with a count of how many dependency paths lead to it - handy for spotting shared utilities buried several levels deep.\nBoth DMOs have their place. Use sys.sql_expression_dependencies for a clean direct-dependency list. Reach for sys.dm_sql_referenced_entities when you need column granularity or cross-database object IDs. And use the recursive CTE when you need the full picture - every object your procedure touches, no matter how deep it's nested.\nTurns out it's not turtles procedures all the way down after all. Or maybe it is - but at least now you can see exactly how far.\n","permalink":"/posts/finding-nested-references/","tags":["Debugging"],"title":"Finding nested references"},{"categories":["T-SQL Tuesday"],"contents":" T-SQL Tuesday #177 Hosted by Mala Mahadevan Topic: Managing database code I have yet to see a perfect implementation but even a partial one benefits you. My experience is mostly with enterprise-scaled environments - large servers, minimal downtime, brownfield development, mostly single-tenant.\nBut these concepts and building blocks should be generic enough.\nThroughout this article, I'll use the word schema. It's an overloaded term but in this case, I'm referring to a general \"shape\" of the database (object definitions with their results sets, columns, etc.) and not the schema returned by sys.schemas.\nWith that out of the way, everything can be summed up by this meme:\nSource of Truth You can't handle the source of Truth — Tom We should take inspiration from application development because it's very similar. Database code is still code.\nIt's just a tiny bit harder because you have to persist the state, so you can't just drop the DB and rebuild it from scratch for any change.\nIf you want automatic deployments, you have to store your scripts somewhere. The hint is in the name: source control should store source code.\nAnd when I say source control I mean git. Even though no one understands git, you'll soon enough learn the golden path scenario of about 5-6 commands. And if it doesn't work, you'll nuke the branch and start again until it does (it's the only way to be sure).\nYou should strive to:\nMove the source of Truth from the production database to the source code (no half-measures) Create an empty environment just from your source-controlled scripts This will get you one step away from developing on a shared environment and one step closer to isolation How should I version my scripts? There are two main schools of thought: state-based and migration-based (or a hybrid). In the end, even the state-based approach generates migration scripts to be applied.\nState-based approach You define the start and end state (e.g. script out the table as is and how it should look in the end) Usually, a schema compare tool generates the migration for you (e.g. alter table… add column) Comparing different schemas (dev \u003c-\u003e test or dev \u003c-\u003e prod) will produce different migration scripts State-based approach is great for collaboration You can see the desired state Since source code is usually stored as one script file per object, you can see git conflicts when two people try to edit at the same time The migration scripts are only as good as the schema compare tool. It often struggles with renames, online deployments or anything beyond AdventureWorks-level complexity. Migration-based approach Mostly the opposite of the state-based approach You write numbered migration scripts and apply them in order You store a version table on the target and only apply the missing migrations Better control over deployment script (manually created) Unless you run the migration scripts against an environment, you don't see the desired state Conflicts are harder to detect as changes against the same object can be in different script files Possible out-of-order merges and deploys Hybrid approach The best of both worlds state-based approach for source control migration-based approach for deployments More moving parts mean it's more brittle In the end, it's all migration scripts and you can switch back and forth between the approaches to some extent.\nAnd the rest There is much more to cover. From the top of my head:\ngit branch strategy monorepo vs polyrepo Store application code along with the database code vs separate storage Store all databases in a single repo vs one repo per DB Variations of the above You have to pick your poison either scaling or dependency hell Unit testing Hotfixes and schema drift Empty schema vs minimal dataset vs curated dataset vs generated data Online deployments Rollback vs rollforward Deployment feature flags Free tools, paid tools, homegrown tools Monitoring and alerting This is a genuinely messy topic. There is no silver bullet yet, and the tools and approaches keep evolving.\n","permalink":"/posts/database-ci-cd-basics/","tags":["CI/CD"],"title":"Database CI/CD Basics (T-SQL Tuesday #177)"},{"categories":["Deep Dive"],"contents":"This post is dedicated to all 10 other DBAs that use Service Broker (you know who you are).\nThe main reason for this blog post is that I've got no hits on Google for CSbRollbackHandlerTask::DisableQ.\nTo improve my SEO I'll be the first to write about it (repeatedly) and get that sweet sweet Service-Broker-questions traffic.\nThe problem I was paged about a blocking chain where the blocked resource was a Service Broker queue, the lead blocked transaction was called CSbRollbackHandlerTask::DisableQ, the lock mode was SCH-M and even lock partitioning was involved. SCH-M is incompatible with every other lock mode, so any session trying to read from or activate the queue has to wait. I won't repro fully this time, but I've covered lock partitioning repro in my post on async stats update blocking. The goal was to capture an instance of transaction CSbRollbackHandlerTask::DisableQ with the same lock mode.\nRepro Let's create some Service Broker objects\nCREATE DATABASE ServiceBrokerDemo USE ServiceBrokerDemo GO -- Create the message type CREATE MESSAGE TYPE SB_MessageType VALIDATION = WELL_FORMED_XML GO -- Create the contract CREATE CONTRACT SB_Contract ( SB_MessageType SENT BY INITIATOR ) GO -- Create the target queue and service CREATE QUEUE SB_TargetQueue GO CREATE SERVICE SB_TargetService ON QUEUE SB_TargetQueue (SB_Contract) GO We'll need to get the object_id of the SB_TargetQueue to plug into the monitoring\nSELECT OBJECT_ID('SB_TargetQueue') The value 901578250 in the XE session below is from my instance - yours will differ, so run this first and swap in your result.\nTo monitor, I'll create an Extended Event session CSbRollbackHandlerTask\nCREATE EVENT SESSION CSbRollbackHandlerTask ON SERVER ADD EVENT sqlserver.broker_queue_disabled ( SET collect_database_name = 1 ACTION ( sqlserver.is_system ) ) , ADD EVENT sqlserver.lock_acquired ( SET collect_database_name = 1 , collect_resource_description = 1 ACTION ( sqlserver.is_system ) WHERE [object_id] = 901578250 /* insert Queue's object_id here */ AND [mode] = 'SCH_M' ), ADD EVENT sqlserver.sql_transaction ( ACTION ( sqlserver.is_system ) WHERE [object_name] LIKE N'CSb%' ) GO ALTER EVENT SESSION CSbRollbackHandlerTask ON SERVER STATE = START Note: XE uses SCH_M (with an underscore) in the mode predicate; in prose and DMV output you'll see it written as SCH-M.\nTest manual disable Can it be as simple as disabling the queue?\nALTER QUEUE SB_TargetQueue WITH STATUS = OFF The SCH-M lock is there (as expected, I'm disabling the queue) but no luck with the CSbRollbackHandlerTask::DisableQ transaction.\nThe difference: manual ALTER QUEUE ... STATUS = OFF is a user DDL transaction, so is_system comes back False. Poison message handling fires under an internal system task - and that distinction matters, as we'll see next.\nTest poison message The only other way to automatically disable a queue (I know of) is via poison message handling.\nAfter I fail to process a message 5 times it gets marked as poisoned and the queue gets disabled (a safeguard so a failing message doesn't block the queue forever).\nSo I'll open up two sessions - one to send a message and another one where I'll repeatedly fail to process it.\nDECLARE @conversation_handle uniqueidentifier DECLARE @message_body xml BEGIN DIALOG @conversation_handle FROM SERVICE SB_TargetService TO SERVICE N'SB_TargetService' ON CONTRACT SB_Contract WITH ENCRYPTION = OFF SELECT @message_body = N'\u003cSEO\u003eCSbRollbackHandlerTask::DisableQ\u003c/SEO\u003e' SEND ON CONVERSATION @conversation_handle MESSAGE TYPE SB_MessageType (@message_body) And in another session receive it in an infinite loop where I'll rollback each time and exit the loop on no messages processed. Don't forget to reenable the queue after Test 1 ALTER QUEUE SB_TargetQueue WITH STATUS = ON.\nDECLARE @conversation_handle uniqueidentifier DECLARE @message_body xml WHILE 1 = 1 BEGIN BEGIN TRAN WAITFOR ( RECEIVE TOP(1) @conversation_handle = conversation_handle , @message_body = CAST(message_body AS XML) FROM SB_TargetQueue ) IF (@@ROWCOUNT = 0) BEGIN ROLLBACK BREAK /* break out of the loop on error */ END ROLLBACK /*5 rollbacks in a row trigger poison message handling */ END After five runs, I get an error message\nMsg 9617, Level 16, State 1, Line 13\nThe service queue \"SB_TargetQueue\" is currently disabled. Looking at the XE session\nI get both sql_transaction with the correct name - CSbRollbackHandlerTask::DisableQ and a broker_queue_disabled event. Unlike the manual disable, both entries have is_system = True - this is an internal system task, not a user transaction.\nThis is also why the blocked process report is useless here: when it detects a system process on either end, it stops giving useful details. The sad part is that it considers an Activation Stored Proc a background process and refuses to give me that information either. This and everything else is why debugging Service Broker issues isn't fun.\nCSbRollbackHandlerTask::DisableQ\n","permalink":"/posts/service-broker-blocking/","tags":["Blocking","Debugging","Locking"],"title":"Service Broker Blocking"},{"categories":["Tools"],"contents":"No matter how hard Azure Data Studio (ADS) is pushed by Microsoft, most DBAs still use SQL Server Management Studio (SSMS). Here is my initial setup, plus the tips and tricks I lean on daily.\nI'm also a loyal user of SQL Prompt by Redgate, so while you might see some of its effects in the following screenshots, I'll probably cover it in a separate post. With that out of the way, let's get to it.\nLayout I'd like to pin my Object Explorer and Registered Servers to the right and set them to auto-hide for these reasons:\nI spend most of my time in the Query window and I like that it's left-aligned I have to resize the Object Explorer based on the level of nesting or length of the object name, SQL jobs, etc The Solution Explorer in Visual Studio is also on the right side 1 Same thing for the Registered Servers 2 Object Explorer is usually below Registered Server but I have no hard preference 3 You might have also noticed that I have the Extended Events toolbox permanently pinned. It opens automatically when using Extended Events views, but I don't like the layout change if I'm switching between the windows Tools\\Options Just keep in mind that after you change an option (whether you enable or disable it), it only applies to new tabs.\nEnvironment Keyboard I have some custom shortcuts, but I will cover them in a separate section near the end.\nQuery Shortcuts I personally only use the default Alt+F1 to run sp_help. I know other people are using all the available slots (and more power to them), but I like the more versatile SQL Prompt snippets.\nStartup Switch to Open empty environment. It makes starting a new instance a bit faster.\nTabs and Windows Insert new tabs to the right of existing tabs - Helps you preserve the ordering of your tabs\nShow pinned tabs in a separate row - while the normal tab row hides tabs if you have more tabs than the window width, the pinned tabs wrap around. 1 The first two rows from the top are the pinned tabs - they wrap around 2 The last row is the normal tab. You can see SQLQuery43 - 46 are missing because they are scrolled out of view. Projects and Solutions I don't use them. I've tried before but didn't see the benefit.\nText Editor Transact-SQL General Word wrap - enable along with Show visual glyphs for word wrap\nYou can see the glyph at the end of a line, also notice that the next \"line\" doesn't have a line number. I usually have two windows side-by-side so this helps for longer lines of code.\nLine numbers - enable. To be honest, I'm not sure why this is not enabled by default\nScroll Bars I always enable Map mode; the only decision is how wide to set it. It depends on the amount of real estate you're willing to give up.\nI usually set it to wide anyway and there is still plenty of space on my 32\" monitor even when having two vertical tab groups.\n1 You'll get a vertical split screen button. Just drag and drop This is useful if you need to cross-check code that is apart (e.g. temp table definition and insert) If you want to cancel the split, just double-click the splitting line. The split screen part that had focus will be kept. 2 Bookmarks are shown as black rectangles (if you're using them) 3 Errors show up as red rectangles If you're not happy with the map mode, there is a shortcut to the settings, simply right-click the scrollbar. Tabs I won't get into the whole tabs vs spaces war (because spaces are superior)\nSet the tab size to whatever you prefer (I like 4) and switch to Insert spaces\nThis means you can still press the Tab key but it will produce X spaces. This preference is due to sharing code (blogging, gist, stack overflow, etc.) where spaces are better for the WYSIWYG experience. Editor Tab and Status Bar For Status Bar Content, I keep everything enabled and set execution time to Elapsed. If you want to know the start time then make a habit of writing SELECT SYSUTCDATETIME() before your query.\nTab Text This one is disappointing because there are only 4 settings you can choose, but you cannot reorder them, so most of the text gets cut off. You can only see the full text on hover.\nThe format is: {FileName} - {Server}.{Database} ({Login ({SPID})})\nAs you can see - most of the time you'd only see the file name and login\nFile name - I can't remove this because it's useful when it's a saved file (like 00_RunThisFirst.sql) but when it's an ad-hoc query, it's useless\nI thought I could at least remove the Login. I almost always use the same login - mine. So cutting this makes perfect sense, right? Well the SPID is tied to the login for some reason. You remove login, you remove SPID. And SPID is useful (blocking, KILL command, tracking the status of index rebuild, etc.)\nAnd Server and Database info is usually cut off anyway.\nQuery Execution SQL Server \\ General I don't remember if it's the default, but SET TEXTSIZE: should be the biggest number (it will automatically round to max integer).\nBatch separator - Keep it to GO unless you want to prank someone. Yes, GO is NOT a transact SQL statement, but it's a setting of the client (SSMS) to mark the end of the batch (for example creating a Stored Proc must be the only statement in the batch)\nI'll skip the Advanced and ANSI settings. I don't like to rely on the SSMS setting which is invisible, I rather declare the ANSI settings in my script if needed. For more info, Erik Darling (Darling Data) covers the details in The Art Of The SQL Server Stored Procedure: ANSI/SET Options.\nQuery Results \\ SQL Server Results to Grid / Text Definitely check Retain CR/LF on copy or save - especially useful when copying definitions from sys.sql_modules\nDiscard results after execution - It's useful if you want to measure the query performance without rendering potentially large output. I've experimented with setting this only for Results to text and then toggling between grid and text results for quick discard. But I don't use it that often.\nMaximum Characters Retrieved - set max int for non-XML data and Unlimited for XML data to avoid truncation of large results.\nMultiserver Results This applies if you open a connection to a group of servers (from the Registered Servers window) I leave defaults on, but it's a matter of preference. Just be aware that it exists.\nDesigners Don't use them. Get in the habit of writing TSQL scripts. They give you better control and can be put into source control or shared.\nSQL Server Object Explorer Scripting Again be aware of this. There are a couple of options that are NOT enabled by default (and IMHO should be)\nData Compression Options Extended properties Permissions Collation Partition schemes Triggers And that's it for the Options section.\nHotkeys Nothing will improve your productivity like knowing hotkeys. I'll skip the standard ones like CopyPaste, etc.\nThese are my essentials\nQuery execution Ctrl+E - Execute Ctrl+R - Results (Toggle between show and hide) Ctrl+L - Display Estimated Execution Plan (doesn't run the query but shows the plan immediately) Ctrl+M - Include Actual Execution Plan (this is a toggle, so it shows the actual plan on next run) Ctrl+D - Results to Grid (default) Ctrl+T - Results to Text. Mostly activated by accident when I meant to press Ctrl+R F4 - (Not to be confused with Alt+F4). Opens the Properties windows - especially useful when looking at execution plans. Sometimes you have to click on another plan node and back to get the window to refresh. Query writing F1 - Highlight a function or DMV and press this hotkey to open documentation (if available) Alt+F1 - Runs sys.sp_help with the highlighted object as the runtime value Ctrl+Spacebar - brings up the Intellisense if you accidentally close it Ctrl+Shift+Spacebar - when inside the parenthesis, it brings up the parameter info window Ctrl+Shift+U - Turn selected word to uppercase Ctrl+Shift+L - Turn selected word to lowercase Alt as a modifier Alt+Up / Alt+Down - moves the current line up or down. Great for changing column order (when each column has its own line) Alt + mouse drag - multi-line edits (not as good as VS Code's multi-cursor, but good enough) Navigation Ctrl+) - If you have caret on a parenthesis or begin/end it will take you to the matching pair (this might be a regional keyboard thing, so check the shortcut in keyboard settings. It's named Edit.GotoBrace) Ctrl+G - GoTo line number Ctrl+Alt+G - Open Registered Servers F8 - Open Object Explorer F7 - Open Object Explorer Details - I thought this was neat back when I was starting out, but haven't opened it in years. They are next to each other, so I execute a query and hide or show Results as necessary. I don't use F5 much, usually on the Shift+F5 variant from SQL Prompt that runs only the current statement.\nCustom hotkeys Window.QuickLaunch - Ctrl+Q - I was surprised this was not default as is the case with Visual Studio. Window.NewVerticalTabGroup - Ctrl+Alt+T - this splits the screen vertically so I can move tabs around. I usually have them for comparison, or one half of the screen is for the project and the other is for ad hoc queries. Window.MovetoPreviousTabGroup - Ctrl+Alt+Left Window.MovetoNextTabGroup - Ctrl+Alt+Right Window.PinTab - Ctrl+Alt+Q - The Q kinda looks like a pin in some fonts Window.PreviousTab - Ctrl+Alt+PgUp. This one is actually on by default but the shortcut in SSMS is misleading as it omits the Alt Window.NextTab - same as above Miscellaneous tips There was a time when the Cycle Clipboard Ring was useful. But now that Windows clipboard has a history, I always use Win+V If you're trying to find a specific item in the Object Explorer use the filter icon on the parent node (e.g. Tables, SQL Agent jobs, etc.) before expanding them. You can toggle the visibility of whitespace characters. It might be handy to check if no one inserted tabs or left trailing whitespace. There are probably many more tips that I didn't recall yet, but this blog post is long enough already. Let me know in the comments if there's something I've missed or that you'd like to see in a future post.\n","permalink":"/posts/my-toolbox-ssms/","tags":["Productivity"],"title":"My Toolbox - SSMS"},{"categories":["Deep Dive"],"contents":"I recently encountered an issue where an index rebuild set to wait_at_low_priority ended up blocking an asynchronous statistics update. This interaction led to a large blocking chain where queries were waiting on the async stats update and started to timeout.\nHow did it happen? We'll need a basic understanding of a few concepts as well as database settings:\n'RCSI' and 'Async stats update' database configuration Auto stats update and how it's triggered Lock partitioning Locking and blocking Database configuration RCSI (Read Committed Snapshot Isolation) prevents reading queries from blocking writers - this helps with the concurrency Stats update: Sync stats - the query will wait for the out-of-date statistics to update before generating a plan. This might be a problem if you have a short query with a high call frequency where waiting for the stats update would take much longer than the query itself Async stats - when stats recomputation is needed it will start a background job that will update the stats later. The query is still using the out-of-date stats but at least it can continue Let's create a test database with RCSI and Async stats update:\nCREATE DATABASE StatsUpdateAsync ALTER DATABASE StatsUpdateAsync SET READ_COMMITTED_SNAPSHOT ON WITH ROLLBACK IMMEDIATE ALTER DATABASE StatsUpdateAsync SET AUTO_UPDATE_STATISTICS_ASYNC ON WITH ROLLBACK IMMEDIATE Auto stats update In one of my previous articles KEEP PLAN Demystified I've demonstrated how to monitor the stats update, what are the different thresholds etc., so if you're not familiar with these concepts, I suggest you read it first.\nFor our demo, we'll create a table with 500 rows which means that the threshold for auto stats update is also 500.\nCREATE TABLE dbo.PermaSmall ( n int , CONSTRAINT PK_PermaSmall PRIMARY KEY (n) ) ;WITH L0 AS(SELECT 1 AS c UNION ALL SELECT 1), L1 AS(SELECT 1 AS c FROM L0 CROSS JOIN L0 AS B), L2 AS(SELECT 1 AS c FROM L1 CROSS JOIN L1 AS B), L3 AS(SELECT 1 AS c FROM L2 CROSS JOIN L2 AS B), L4 AS(SELECT 1 AS c FROM L3 CROSS JOIN L3 AS B), L5 AS(SELECT 1 AS c FROM L4 CROSS JOIN L4 AS B), Nums AS(SELECT ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) AS n FROM L5) , tally AS (SELECT TOP (500) n FROM Nums ORDER BY n) INSERT INTO dbo.PermaSmall WITH(TABLOCKX) (n) SELECT n FROM tally Lock partitioning This was the hardest piece of the puzzle because there is not a whole lot of documentation.\nMy main sources were:\ndocumentation for SQL 2008 Bob Dorr's article How It Works: SQL Server Lock Partitioning Understand lock partitioning A previous email conversation I had with Jonathan Kehayias . I'll summarize the key points:\nLock partitioning is enabled by default if you have 16+ CPUs Startup Trace Flag 1228 can lower the requirement to 2 CPUs You can check if it's enabled by looking for the message Lock partitioning is enabled… in the Error Log. For example with this query EXEC sp_readerrorlog 0, 1, N'lock partitioning' It's a scalability feature when some types of lock can be split into partitions enabling higher concurrency for some operations The number of partitions matches the number of schedulers Acquiring shared access requires only the local partition to be acquired Acquiring exclusive access requires all partitions to be acquired Partition lock acquires always start at 0 and go up to the number of schedulers. Lock releases have reversed direction Locking and blocking This should not be a new concept to most DBAs. The main tool to help with this is the Lock compatibility matrix (the complete one).\nIn this demo - the most important locks will be Sch-M (schema modification) and Sch-S (schema stability).\nSchema is an overloaded term that can mean multiple things. In this case, it refers to the \"definition\" of an object.\nSch-S lock is present in every query because you don't want the schema to change inflight. This lock cannot be removed (yes, not even with the NOLOCK hint). Sch-M lock on the other hand blocks everything and must wait on the release of all the Sch-S locks so you can for example add a new column or rebuild an index, etc. Demo Control the assigned scheduler We're almost ready to start the demo, but there is one more consideration that caused me a headache. In the Lock partitioning summary, we had this bullet point:\nAcquiring shared access requires only the local partition to be acquired\nThe local partition matches the scheduler_id that the session is running on. If we want a repeatable demo, we must be able to control the session's scheduler assignment. After some attempts with closing and opening new sessions hoping for the desired scheduler, I've opted to use the Resource Governor's AFFINITY option.\nThis snippet will dynamically create Resource pools with workgroups per scheduler and a Classification function that will match the workgroup to the application name.\nDECLARE @schedulerCount int SELECT @schedulerCount = dosi.scheduler_count - 1 /* zero based */ FROM sys.dm_os_sys_info AS dosi DECLARE @i int = 0 , @dynamicSql nvarchar(MAX) = N'' WHILE @i \u003c= @schedulerCount BEGIN SET @dynamicSql = N' CREATE RESOURCE POOL SchedulerPool' + CAST (@i AS varchar(2)) + N' WITH (AFFINITY SCHEDULER = (' + CAST (@i AS varchar(2)) + N'));' EXECUTE sp_executesql @dynamicSql SET @dynamicSql = N' CREATE WORKLOAD GROUP SchedulerGroup' + CAST (@i AS varchar(2)) + N' USING SchedulerPool' + CAST (@i AS varchar(2)) + ';' EXECUTE sp_executesql @dynamicSql SET @i = @i + 1 END GO CREATE OR ALTER FUNCTION dbo.ClassifierFunction() RETURNS SYSNAME WITH SCHEMABINDING AS BEGIN DECLARE @WorkloadGroup SYSNAME IF APP_NAME() LIKE 'SchedulerGroup%' /* example: SchedulerGroup15*/ BEGIN SET @WorkloadGroup = APP_NAME() END ELSE BEGIN SET @WorkloadGroup = 'default' END RETURN @WorkloadGroup END GO ALTER RESOURCE GOVERNOR WITH (CLASSIFIER_FUNCTION = dbo.ClassifierFunction) GO ALTER RESOURCE GOVERNOR RECONFIGURE GO Now if I want my SSMS session to run on scheduler 6 I'll add an Additional Connection parameter\nApplication Name=SchedulerGroup6; And I can check that it worked with:\nSELECT dot.scheduler_id , dot.session_id FROM sys.dm_os_tasks AS dot WHERE dot.session_id = @@spid Plan of action To demonstrate the blocking, I will:\nStart a reading query in a transaction with HOLDLOCK. I will only read the last row in the table Start an index rebuild with WAIT_AT_LOW_PRIORITY and a long timeout so we have time to test this This will be blocked because of the Sch-S lock from the reading query To demonstrate the blocking, this should be running on a scheduler with a high number (let's say 10) Modify the table a couple of times to bring the stats modification counter over the threshold This is not blocked by the reading query as I'm only reading the last row It's also not blocked by the index rebuild which waits in the low-priority queue Run a query reading from the table - this will trigger the Async update stats To demonstrate the blocking we can run the reading query again from a session lower than 10 Blocked rebuild Let's run the first few steps:\nRun this query in the StatsUpdateAsync database - the scheduler_id doesn't matter yet\nUSE StatsUpdateAsync GO /* Blocker */ BEGIN TRAN SELECT * FROM dbo.PermaSmall AS ps WITH (HOLDLOCK) WHERE ps.n = (SELECT 500) -- ROLLBACK In another session with the app name Application Name=SchedulerGroup10 run\nSELECT dot.scheduler_id , dot.session_id FROM sys.dm_os_tasks AS dot WHERE dot.session_id = @@spid USE StatsUpdateAsync GO /* Rebuild */ ALTER INDEX [PK_PermaSmall] ON [dbo].[PermaSmall] REBUILD WITH ( ONLINE=ON ( WAIT_AT_LOW_PRIORITY ( MAX_DURATION = 50 MINUTES /* I don't mind waiting */ , ABORT_AFTER_WAIT = SELF ) ) ) The query will be stuck in executing and we have 50 minutes to run our test.\nNow we'll run the update query twice and check the modification counter\nUSE StatsUpdateAsync GO /* Updater */ UPDATE p SET p.n = p.n /* fake modification */ FROM dbo.PermaSmall AS p WHERE p.n \u003c= 250 OPTION (KEEPFIXED PLAN) SELECT obj.name AS tableName , stat.name AS statName , CAST(sp.last_updated AS time) AS Last_update_time , sp.rows , sp.steps , sp.modification_counter AS modCounter , d.compatibility_level AS CL , 500 AS threshold FROM sys.objects AS obj JOIN sys.stats AS stat ON stat.object_id = obj.object_id JOIN sys.stats_columns AS sc ON sc.object_id = stat.object_id AND sc.stats_id = stat.stats_id AND sc.stats_column_id = 1 JOIN sys.columns AS c ON c.object_id = obj.object_id AND c.column_id = sc.column_id CROSS APPLY sys.dm_db_stats_properties (stat.object_id, stat.stats_id) AS sp JOIN sys.databases AS d ON d.database_id = DB_ID() WHERE obj.is_ms_shipped = 0 AND obj.name LIKE N'Perma%' ORDER BY sp.rows OPTION (RECOMPILE, KEEPFIXED PLAN) Now by running a reading query once we trigger the async auto stats update.\nLet's take a look at the locks:\nSELECT dtl.request_session_id AS spid , CASE WHEN deib.event_info IS NOT NULL THEN LEFT(deib.event_info, 20) WHEN debjq.object_id1 IS NOT NULL THEN 'Async stats update' ELSE 'N/A' END AS Spid_Info , dtl.resource_lock_partition AS lock_partition , dot.scheduler_id , dtl.resource_type AS type , dtl.resource_subtype AS subtype , dtl.request_mode AS lock , dtl.request_status AS status , dtl.resource_description , dtl.resource_associated_entity_id , dtl.request_lifetime , dtl.request_owner_type , dtl.lock_owner_address , debjq.session_id FROM sys.dm_tran_locks AS dtl JOIN sys.dm_os_tasks AS dot ON dtl.request_session_id = dot.session_id LEFT JOIN sys.dm_exec_background_job_queue AS debjq ON debjq.session_id = dtl.request_session_id OUTER APPLY sys.dm_exec_input_buffer(dtl.request_session_id, 0) AS deib WHERE dtl.resource_database_id = DB_ID('StatsUpdateAsync') AND dtl.request_mode LIKE 'Sch-%' AND dtl.resource_type = 'METADATA' AND dtl.resource_subtype = 'STATS' ORDER BY dtl.request_session_id, dtl.resource_lock_partition The Rebuild task is running on scheduler 10 and is holding Sch-S lock only on the local partition (same as the scheduler_id) The Async stats update acquired Sch-M locks on partitions 0 - 9 but because modify has to lock all partitions, it's blocked by the incompatible Sch-S held by the Rebuild Now any query running on scheduler_id less than 10 will be blocked by the most restrictive Sch-M lock of the async stats update But any query running on scheduler_id greater than 10 can still continue as usual You can test yourself by repeating the Reader or Updater query on any of the schedulers using the appropriate app_name Warning I had inconsistent results when trying to run the Reader on the same scheduler as the index rebuild. Sometimes it caused a deadlock where the Async stats update was a victim - thus releasing the Sch-M locks and unblocking everyone. But at the same time, it seemed like it queued another Async stats update causing another blocking chain for future queries. Solutions Microsoft is aware of this problem and it has been fixed in the SQL 2022 (and the Azure offerings) as highlighted in this blog post Improving concurrency of asynchronous statistics update\nYou can enable this database-scoped configuration (it's not enabled by default)\nALTER DATABASE SCOPED CONFIGURATION SET ASYNC_STATS_UPDATE_WAIT_AT_LOW_PRIORITY = ON Then the lock information looks like this\nThis is not blocking anymore so queries can continue as usual.\nAs for those not on SQL 2022 yet - our solution was to update stats manually just before the planned index rebuild to minimize the chance of this happening.\nIt also helps if you don't have long-running queries that can block the index rebuild and auto stats update.\n","permalink":"/posts/async-stats-update-causing-blocking/","tags":["Blocking","Debugging","Performance","SQL Server 2022"],"title":"Async stats update causing blocking"},{"categories":["Deep Dive"],"contents":"In SQL Server, using the KILL command to terminate a session results in an entry being logged in the error log. This raises the question: Does the ALTER INDEX REBUILD command with the WAIT_AT_LOW_PRIORITY option also log its actions in the error log?\nKILL demo Let's start with a KILL example. I'll open two connections which I'll refer to as command and victim - note the victim's session ID (SELECT @@SPID). For me, it's 52\nThen simply run KILL 52 from your command session.\nYou can then read and filter the error log with this command\nEXEC sp_readerrorlog 0, 1, N'kill' And this is the output. WAIT_AT_LOW_PRIORITY demo For the second demo, we'll:\nCreate a table with a named PK Fill it with some rows Start a reading transaction in the victim session Run ALTER INDEX REBUILD with WAIT_AT_LOW_PRIORITY and ABORT_AFTER_WAIT = BLOCKERS Check error log again to see if it was logged DROP TABLE IF EXISTS dbo.TestTable CREATE TABLE dbo.TestTable ( Id int , Filler char(100) , CONSTRAINT PK_dbo_TestTable PRIMARY KEY CLUSTERED (Id) ) ; -- Previous statement must be properly terminated WITH L0 AS(SELECT 1 AS c UNION ALL SELECT 1), L1 AS(SELECT 1 AS c FROM L0 CROSS JOIN L0 AS B), L2 AS(SELECT 1 AS c FROM L1 CROSS JOIN L1 AS B), L3 AS(SELECT 1 AS c FROM L2 CROSS JOIN L2 AS B), L4 AS(SELECT 1 AS c FROM L3 CROSS JOIN L3 AS B), L5 AS(SELECT 1 AS c FROM L4 CROSS JOIN L4 AS B), Nums AS(SELECT ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) AS n FROM L5) , tally AS (SELECT TOP (5000) n FROM Nums ORDER BY n) INSERT INTO dbo.TestTable WITH (TABLOCKX) (Id, Filler) SELECT n , CAST(n AS char(100)) FROM tally In the victim session, open a new reading transaction. Note the session ID - for me it's 66, which is the one we'll spot highlighted in the log shortly.\nBEGIN TRAN SELECT * FROM dbo.TestTable (HOLDLOCK) Back in the command session we'll run the ALTER INDEX. This will run for a minute.\nWarning ABORT_AFTER_WAIT = BLOCKERS force-kills every transaction blocking the rebuild. It requires the ALTER ANY CONNECTION permission - without it you'll get error 11423. ALTER INDEX [PK_dbo_TestTable] ON [dbo].[TestTable] REBUILD WITH ( ONLINE=ON ( WAIT_AT_LOW_PRIORITY ( MAX_DURATION = 1 MINUTES , ABORT_AFTER_WAIT = BLOCKERS ) ) ) We can check the error log with the same query as before and the result is:\nConclusion Yes, it's logged - and in more detail than KILL. Where KILL writes a single entry, ABORT_AFTER_WAIT = BLOCKERS generates three: the ALTER INDEX REBUILD start, a lock request (with database_id and object_id), and the killed process ID. That's enough to trace both the victim and the statement that caused it.\nNote I only tested ABORT_AFTER_WAIT = BLOCKERS here. SELF (which aborts the ALTER INDEX itself) and NONE (wait indefinitely) may log differently. ","permalink":"/posts/are-abort_after_wait-victims-logged/","tags":["Debugging","Blocking"],"title":"Are ABORT_AFTER_WAIT's victims logged?"},{"categories":["T-SQL Tuesday"],"contents":" T-SQL Tuesday #166 Hosted by Grant Fritchey (Scary DBA) Topic: Extended Events Foreword My first public speaking session (and so far the only one) was on Troubleshoot Real-World Scenarios with Extended Events. Check the Tags section of this blog and Extended Events sits near the top. I am a fan.\nWhy am I a fan? First, I won't get into the Profiler vs. Extended Events debate. Profiler was already deprecated when I got myself into the database business, so I never had to work with it. People say it has a better GUI, but there is no more development. It's time to move on.\nOnce you reach a certain level of problem complexity, you'll want to look under the SQL Server's hood. And for live event capture, we, mere mortals, have access to only a limited number of tools. I can think of only Trace Flags, DBCC commands, and Extended Events.\nAnd if you have to debug something in production, Extended Events is the safest choice.\nWhat can it do? I'll again refer to the Extended Events tag - there are some interesting problems I could debug only with XE's help, including tracking down production errors. I also have a GitHub repo sharing a few useful code snippets.\nThere are several thousand XEs, and most of them are situational. You'll regularly use about 20 of them (and I'm being generous). But you'll know where to look when the real tricky problem rears its head.\nI'm using this snippet to look for new events. For example, if I want to track why Query Store is switching to read-only - I'll search for Query Store related XEs.\nSELECT dxp.name AS packageName , dxo.name AS eventName , dxo.description AS eventDescription , dxoc.name AS columnName , dxoc.column_id , dxoc.type_name , dxoc.description AS columnDescription , ca.map_agg , dxoc.column_type , dxoc.capabilities_desc , sum (IIF(dxoc.description IS NULL, 1, 0)) OVER () AS NullDescriptionCount , sum (IIF(dxoc.description IS NOT NULL, 1, 0)) OVER () AS NotNullDescriptionCount FROM sys.dm_xe_objects dxo JOIN sys.dm_xe_packages dxp ON dxo.package_guid = dxp.guid JOIN sys.dm_xe_object_columns AS dxoc ON dxoc.object_name = dxo.name AND dxoc.object_package_guid = dxo.package_guid AND dxoc.column_type \u003c\u003e N'readonly' CROSS APPLY ( SELECT STRING_AGG(CAST(dxmv.map_value AS nvarchar(MAX)), N', ') WITHIN GROUP (ORDER BY dxmv.map_key) AS map_agg FROM sys.dm_xe_map_values AS dxmv WHERE dxmv.object_package_guid = dxoc.object_package_guid AND dxmv.name = dxoc.type_name ) AS ca WHERE dxo.object_type = 'event' AND dxo.name LIKE '%query_store%' --AND dxoc.name LIKE '%query_hash%' --AND ca.map_agg LIKE '%abort%' --AND dxp.name = 'sqlserver' You can also uncomment and search in column names, description, map values, etc.\nEvery silver lining has a cloud The tool isn't perfect. The GUI is clunky. You have to get good at TSQL xml parsing, and I could go on. In fact I did and sent MS an A4 with common pain points and suggestions for improvements, but I digress.\nOften, I have to write my own procedure to read from the target .xel files, and it usually involves looping through databases, string parsing and trying to get resource names from their IDs.\nIdeally, I'd love a built-in, reliable way of collecting the XE data into a central monitoring storage with pre-processing similar to Query Store's aggregation.\nTip XESmartTarget by Gianluca Sartori (Spaghettidba) is the closest open-source option. If MS ever ships something like this out-of-the-box, the tool will be in a much better place.\nOne can dream... Until then, Extended Events stays at the top of my diagnostic toolkit.\n","permalink":"/posts/extended-events-and-i/","tags":["Extended Events"],"title":"Extended Events and I (T-SQL Tuesday #166)"},{"categories":["Investigation"],"contents":"Foreword The other day I managed to confuse myself. I was looking up some information from an Extended Events (XE) session, but my eyes were playing a trick on me. The database ids were off by one, and I couldn't find some query hashes in the Query Store, even when they were supposed to be there. So my first thought was that I must be connected to a different server with a drift. But the information in the SSMS tab, status bar and even colour coding (courtesy of Redgate's SQL Prompt) - all pointed to the correct server.\nWhen I ran SELECT @@SERVERNAME, I found out I was right.\nHow did it happen? To demonstrate this, I will need two different SQL Servers. I already have a local instance, and I'll use a Docker container for the other one. I keep one around for exactly this kind of sandbox - more on why I run SQL Server in Docker.\nThe SQL Server version doesn't matter; if you locally have a different image, feel free to use that one.\ndocker run ` -e 'ACCEPT_EULA=Y' ` -e 'SA_PASSWORD=Password5' ` -e 'MSSQL_PID=Developer' ` --hostname bamboozled ` -p 14338:1433 ` -d ` --name tempsql ` mcr.microsoft.com/mssql/server:2019-latest Warning Password5 is a throwaway password for a local sandbox container only. Don't use it on anything real. Take note of the port number 14338 and the hostname.\nTo connect there, I'll open a New Query window and fill in all the necessary information like so:\nAnd running the previous query, we can see that everything matches the information from the docker run command and the colour coding of the tab and status bar is yellow (that's how I denote a sandbox environment).\nSo far, no surprises there.\nBut the connection window has an Options button, revealing additional tabs. For example, on the Connection Properties tab, you can specify a default database for a connection or a shorter timeout (especially useful for the Docker containers). But I'm interested in the last tab, called Additional Connection Parameters.\nI usually use this tab to specify an Application Name parameter. Then I can use the application name to filter XE or Adam Machanic 's sp_WhoIsActive program_name column (instead of another boring Microsoft SQL Server Management Studio - Query).\nApplication Name=It's a me, Mario; But at the very bottom of this tab, there is a note that gives away the solution. Emphasis mine:\nNote Connection string parameters override graphical selections on other panels Meaning, if I put a whole connection string there, like:\nServer=localhost,14338;Database=master;User ID=sa;Password=Password5;Trusted_connection=False; It will connect to the Docker container, no matter what I've selected on the first tab.\nWhat really happened I was aware of this behaviour, so how did I confuse myself? When you have a focus in an existing tab and click on a New Query - it copies the whole deal, including the Additional Connection Parameters, so when you click on the Change Connection and only change the server in the graphical menu, it still gets overridden.\nSo make sure you keep these special connections contained, so you don't accidentally copy them.\nWhat is it good for? Absolutely nothing! Maybe it's a cautionary tale or a harmless prank you can play on your colleague when April Fool comes around. Just make sure you're not connecting to Production!\nAs always, thanks for reading.\n","permalink":"/posts/misleading-ssms-connection/","tags":["Debugging","Docker"],"title":"Misleading SSMS Connection"},{"categories":["Non-technical"],"contents":"Foreword Most people solve Advent of Code in Python. I'm doing it in SQL. And then, because I've started learning KQL, I'm solving each puzzle a second time in KQL too.\nAdvent of Code is an annual event in which participants solve a series of coding puzzles. It typically begins on December 1 and runs through the end of the month, with a new challenge being released each day. You can find its homepage here: Advent of Code.\nI've participated in previous years as well, but the difficulty does tend to ramp up as the event goes on, so it's common to have trouble with the later puzzles. The holiday season is also a busy time for many people, so finding the time and focus can be challenging.\nWhat's new Since I've recently started to learn KQL, I've tried solving problems in both languages (SQL and KQL). Once I get to a solution, I usually reuse the algorithm, but sometimes I could use the language's features to my advantage.\nWhere's the code? I'm posting my code to this GitHub repo: zikato/AdventOfCode2022. Don't expect best practices there, I usually race to a correct solution and don't go back for refactor.\nI'm already a bit behind schedule, and I can only expect the difficulty to increase. However, I'll be happy if I can make the first ten challenges.\nIf you have extra time, I encourage you to find solutions independently - it's good practice.\n","permalink":"/posts/advent-of-code-2022/","tags":["KQL","Personal"],"title":"Advent of Code 2022"},{"categories":["How to"],"contents":" Corrections 2026-06-01: Updated the Docker examples to use MSSQL_SA_PASSWORD instead of SA_PASSWORD. The latter is deprecated in recent SQL Server on Linux releases. Foreword Have you ever wondered where the .xel file is saved when you create a new Extended Event session and don't specify the full path (just the file name)?\nLike so:\nWell, so did I and here's what I've found out.\nTest Let's run our tests on a local instance because we'll have to restart it at some point.\nCREATE EVENT SESSION [TestFileTarget] ON SERVER ADD EVENT sqlserver.sql_statement_completed ADD TARGET package0.event_file ( SET filename=N'TestFileTarget',max_file_size=(2) ) WITH (STARTUP_STATE = ON) I created the session via GUI and then scripted it out. You can see that even the scripted version contains only the file name.\nWe have to start the session if we want to find out through TSQL where the file is saved.\nALTER EVENT SESSION [TestFileTarget] ON SERVER STATE = START That populates the sys.dm_xe_sessions DMV, and we can find the current location with this snippet.\n; -- Previous statement must be properly terminated WITH xeTargets AS ( SELECT s.name , t.target_name , CAST(t.target_data AS xml) AS xmlData FROM sys.dm_xe_session_targets AS t JOIN sys.dm_xe_sessions AS s ON s.address = t.event_session_address ) SELECT xt.name , xt.target_name , xNodes.xNode.value('@name', 'varchar(250)') AS filePath , xt.xmlData FROM xeTargets AS xt /* OUTER APPLY if you want to see other sessions */ CROSS APPLY xt.xmlData.nodes('.//File') AS xNodes (xNode) This is my result; your path will be different:\nLet's look into that folder (use your favourite explorer or cmd line).\nGet-ChildItem -Path \"D:\\SqlServer\\MSSQL16.MSSQLSERVER\\MSSQL\\Log\" | Select-Object Name I can see there are several files:\nerrorlog errorlog.1 HkEngineEventFile_0_133143177576200000.xel log.trc system_health_0_133143177578450000.xel TestFileTarget_0_133143178669430000.xel It seems like it's in the same folder as the errorlog. So let's change the errorlog's path and see if XE event files are also affected.\nThe error log's path is a SQL Server startup parameter. And this is the relevant excerpt from the documentation (emphasis mine).\nIs the fully qualified path for the error log file (typically, C:\\Program Files\\Microsoft SQL Server\\MSSQL.n\\MSSQL\\LOG\\ERRORLOG).\nIf you do not provide this option, the existing registry parameters are used.\nTo change the startup parameter, I'll use Configuration Manager.\nRight-click the SQL Server service, select Properties and then go to the Startup Parameters tab.\nLet's change the folder to something else.\nWarning Provide the full errorlog filename (without extension) at the end of the -e path. If you omit it, SQL Server will not start. I'm changing it to -eD:\\ErrorLog\\Errorlog, a new folder I've created.\nThis change requires SQL Server service restart, so let's do that as well.\nSince we've specified the WITH (STARTUP_STATE = ON), the event is running. So we can check the path with the previously mentioned snippet.\nBut if we check the old errorlog folder, we can see that the old error log and XE files are still there. This is because they are not cleaned up automatically.\nAs you would expect, reading from the XE file shows only the rows stored in the new location, so be mindful of that.\nBonus test If you don't have the SQL Server installed locally, don't worry. We can test it even faster in Docker. I'm using the latest image mcr.microsoft.com/mssql/server:2022-latest but feel free to use a different one if you already have it locally.\ndocker run ` -e 'ACCEPT_EULA=Y' ` -e 'MSSQL_SA_PASSWORD=Password5' ` -e 'MSSQL_PID=Developer' ` -p 14338:1433 ` -d ` --name xefilepath ` mcr.microsoft.com/mssql/server:2022-latest I'll get this result by running the first few steps from the earlier demo (creating and starting the session + finding the file location).\nThat's the default error log path on Linux. But I can change that with an environment variable.\nLet's remove the container and recreate it with the variable MSSQL_ERROR_LOG_FILE.\nNote Same rule as Windows: specify the errorlog filename without the file extension at the end of the path. docker rm -f xefilepath docker run ` -e 'ACCEPT_EULA=Y' ` -e 'MSSQL_SA_PASSWORD=Password5' ` -e 'MSSQL_PID=Developer' ` -e MSSQL_ERROR_LOG_FILE='/var/opt/mssql/dontblink/errorlog' ` -p 14338:1433 ` -d ` --name xefilepath ` mcr.microsoft.com/mssql/server:2022-latest Since we removed the container, we have to recreate the session again. When we do that, we can see that the Extended Event file is created in the expected folder.\nThe takeaway: when you skip the full path, event_file lands wherever the error log lives. Specify a full path and you control it yourself - but if you leave it to the default, it follows the error log location. Change that via Configuration Manager on Windows or MSSQL_ERROR_LOG_FILE in Docker, and your default XE file location moves with it.\nIf you're new to Extended Events sessions, Investigating errors with Extended Events is a practical place to start.\n","permalink":"/posts/default-event_file-path-for-extended-events/","tags":["Extended Events","Docker"],"title":"Default event_file path for Extended Events"},{"categories":["T-SQL Tuesday"],"contents":"Foreword Thank you everyone for participating in November's T-SQL Tuesday! There is a total of 15 submissions and thanks to them, I've widened my perspective.\nThe order of the posts is chosen at random.\nGreg Moore Greg Moore Greg has a shortlist of best practices he wants to have in production code. I especially liked the one about alerting being actionable.\nRead the contribution\nAaron Bertrand Aaron Bertrand Aaron has a list of 5 examples with both the bad and good alternatives. I'm in complete agreement, and I will fail any code review that uses shorthand dateparts.\nRead the contribution\nOlivier Van Steenlandt Olivier Van Steenlandt Olivier has a practical checklist of steps he takes to make the code production worthy. Having a checklist is great, so you don't accidentally forget something.\nRead the contribution\nKenneth Fisher Kenneth Fisher Kenneth's answer is in true DBA fashion: It depends. Read on to find out which criteria affect the answer.\nRead the contribution\nDeepthi Goguri Deepthi Goguri Deepthi has a list of several tips for you. Many of those points relate to the readability of the code, which is extremely important.\nRead the contribution\nDamien Jones Damien Jones (amazonwebshark) Damien went all out and used three different programming languages. Each example has a different scenario, problem and resolution.\nRead the contribution\nAjay Dwivedi Ajay Dwivedi Ajay has a list of practices and other SOLID advice - from naming conventions to testing and tools used to build the project.\nRead the contribution\nDavid Wiseman David Wiseman I agree with everything on David's list. I love the points about which things specifically prevent the code from being production-grade.\nRead the contribution\nJosh Darnell Josh Darnell Josh invites you to think about scalability and maintenance, among other things. Each task can have different criteria based on how vital it is.\nRead the contribution\nDeborah Melkin Deborah Melkin Deborah warns us that there is nothing more permanent than temporary code. Therefore, you should have some level of standards and require it for all the code.\nRead the contribution\nKevin Chant Kevin Chant I like to have fun during work, but I agree with Kevin that there are better mediums for jokes than code comments.\nRead the contribution\nChad Callihan Chad Callihan Chad doesn't let \"perfect\" be the enemy of \"good\" but still has a list of several quality gate requirements.\nRead the contribution\nAndy Yun Andy Yun Andy requires all code to be tested. But there are exceptions to every rule when production is at stake.\nRead the contribution\nRob Farley Rob Farley I caught Rob by surprise with this invitation. But, of course, \"production-grade\" can mean different things to different people. Still, Rob has some advice on where to look for inspiration.\nRead the contribution\nTom Zíka Tom Zíka (StraightforwardSQL) My submission is about the importance of testing. But I wrote the post on the last day of the deadline and had to publish a minimal viable blog post (MVP). So that's an unintended meta-commentary on the production code.\nRead the contribution\n","permalink":"/posts/tsql-tuesday-156-wrap-up/","tags":null,"title":"T-SQL Tuesday #156 - Wrap Up"},{"categories":["T-SQL Tuesday"],"contents":" T-SQL Tuesday #156 Hosted by Tom Zíka (StraightforwardSQL) Topic: Production code Even though I picked the question, I struggled to answer it. Following my train of thought: production code should be of the highest quality. To enforce quality, we use quality gates. And the one I value above all is testing.\nWe want to ensure the code behaves as expected when we deploy it to production. And because the only constant is \"change\", ideally, we want the tests to be automated so the next person that touches the code doesn't have to repeat all that work.\nI'll cover unit tests and their benefits on a regex example, because unit testing SQL code is too big a topic for one article.\nExample My use case was to find static references to procedure names in a C# codebase. To start, I only had a few simple criteria:\nProcs can either have a schema or not All combinations of quoted schema / proc name The schema and proc name are captured without the brackets I'm a fan of Test-driven development because it matches reality. Therefore, I have specifications before I write any code.\nLet's start creating the unit tests. I'm using regex101 and a .NET flavour.\nI need to provide a description and an assertion to create unit tests.\nIdeally, I should add more tests, especially for false positives and values I don't want to capture. This might be a bit of trial and error.\nNext, I'll create a \"simple\" regex that will satisfy the criteria.\n^(?:\\[?(\\w+)]?\\.)?\\[?(\\w+)]?$ You can check the regex, unit tests and the explanation in this regex101 snippet.\nThis will serve as living documentation containing all the use cases. There is also an explanation of that regex on the right-hand side.\nNext time, I can improve the performance or add a new requirement: stored procedures can be versioned using the format #V123 (hash symbol and letter V followed by numbers), and we want to capture this version, because that way we can check if our codebase calls two different versions of the procedure.\nThe new developer can add new unit tests, and if they cause a regression, it's detected instantly during the early phases of development, thus providing a fast feedback loop.\nThat feedback loop is the quality I value most in production code - tests let the next person change it without fear.\n","permalink":"/posts/testing-the-code/","tags":["Regex","CI/CD"],"title":"Testing the code (T-SQL Tuesday #156)"},{"categories":["T-SQL Tuesday"],"contents":"Background T-SQL Tuesday - the brainchild of Adam Machanic and coordinated by Steve Jones is a monthly blog party on the second Tuesday of each month. And I will be your host for November 2022.\nInvitation I'm a learner by example, so when I started programming (not so long ago), I tried to find existing solutions on various Q\u0026A sites or blogs, as one might.\nAfter a while, I noticed one sentence repeating often enough that it stuck with me:\n\"This is not a production-grade code\".\nSo here's my invitation: \"Which quality makes code production-grade?\"\nYou might think: \"Production code is code that runs in production, duh.\"\nBut let's help out the newbies who look for a bit of concrete guidance. Please be as specific as possible with your examples and include your reasoning.\nI'm not limiting the scope to just the SQL; it can be anything.\nRules Publish on Tuesday, 2022-11-08 (any time zone). Use the T-SQL Tuesday logo and link back to this post. Use the #tsql2sday hashtag on Twitter. Leave a comment here with a link to your post. DO NOT talk about Fight Club. Update: the party is long over. Head to the #156 round-up to see what everyone said makes code production-grade.\n","permalink":"/posts/production-code/","tags":null,"title":"Production Code (T-SQL Tuesday #156)"},{"categories":["Deep Dive"],"contents":"Foreword In the previous posts, we covered why Scalar UDFs are bad for parallelism and performance, and what the options are for their removal.\nThat leaves one question: where do you start? There's no single right answer, so instead I offer several strategies you can mix and match.\nOptimize your workload regardless of Scalar functions Open your favourite monitoring tool (mine is Query Store) and find the Top CPU consuming queries in your workload. Optimize as usual, but if you run across a UDF, you probably have an easy win right there.\nUDFs in table and view definitions That means Check constraints and Computed columns.\nWhile these might be harder to replace (sometimes you have to move the logic elsewhere), these will have a significant impact. Because Scalar UDFs are parallelism inhibitors for anything that touches those tables, there is a good chance you want that parallelism enabled.\nUDFs in triggers Similar logic. You'll be paying the UDF tax if your table sees any activity that fires the triggers. I would start with the most frequently accessed tables.\nEasy to rewrite UDFs Maybe you need some quick and easy wins to get people on board with the idea of getting rid of the UDFs. Easy to rewrite can mean several things:\nNot referenced in many objects - so they are easy to replace with fewer deployments. Easily testable. Maybe it has only a few parameters or not many code paths. Easy to convert to an ITVF. It might already only have one statement, and the only change you need is to replace RETURNS [data_type] with RETURNS TABLE. A code snippet later in this post helps with finding UDFs with the lowest number of references.\nTop resource-consuming UDFs A DMV sys.dm_exec_function_stats tracks the aggregated statistics of cached functions. That means a server restart or other activity clears the cache and affects it. Nonetheless, it's where you can find the worst UDF performance offenders across the whole instance.\nSnippets Now that we've covered the strategies, here are a few useful snippets.\nFinding the references DROP TABLE IF EXISTS #UdfReferences CREATE TABLE #UdfReferences ( dbName nvarchar(128), refingObjId int, refingObjType nvarchar(60), refingSchName nvarchar(128), refingObjName nvarchar(128), refingColName nvarchar(128), fnObjId int, fnSchName nvarchar(128), fnName nvarchar(128), is_inlineable bit, inliningStatus varchar(3) ) INSERT INTO #UdfReferences ( dbName, refingObjId, refingObjType, refingSchName, refingObjName, refingColName, fnObjId, fnSchName, fnName, is_inlineable, inliningStatus ) SELECT DB_NAME() AS dbName , ro.object_id AS refingObjId , ro.type_desc AS refingObjType , SCHEMA_NAME(ro.schema_id) AS refingSchName , CONCAT ( OBJECT_NAME(ro.parent_object_id) + N'_' , ro.name ) AS refingObjName /* if constraint, CONCAT with the parent table name */ , COALESCE(cc.name, chkConstraint.colName, dfConstraint.colName) AS refingColName , fno.object_id AS fnObjId , fns.name AS fnSchName , fno.name AS fnName , sm.is_inlineable , IIF(sm.inline_type = 0, 'OFF', 'ON') AS inliningStatus FROM sys.sql_expression_dependencies AS sed JOIN sys.objects AS ro ON ro.object_id = sed.referencing_id LEFT JOIN ( SELECT cc.object_id , c.name AS colName FROM sys.check_constraints AS cc JOIN sys.columns AS c ON cc.parent_object_id = c.object_id AND cc.parent_column_id = c.column_id ) chkConstraint ON chkConstraint.object_id = ro.object_id LEFT JOIN ( SELECT dc.object_id , c.name AS colName FROM sys.default_constraints AS dc JOIN sys.columns AS c ON dc.parent_object_id = c.object_id AND dc.parent_column_id = c.column_id ) dfConstraint ON dfConstraint.object_id = ro.object_id LEFT JOIN sys.columns AS cc /* computed column */ ON sed.referencing_id = cc.object_id AND sed.referencing_minor_id = cc.column_id JOIN sys.objects AS fno ON fno.name = sed.referenced_entity_name AND fno.type = 'FN' JOIN sys.schemas AS fns ON fns.schema_id = fno.schema_id AND fns.name = sed.referenced_schema_name JOIN sys.sql_modules AS sm ON sm.object_id = fno.object_id SELECT * FROM #UdfReferences AS ur WHERE ur.refingColName IS NOT NULL OR ur.refingObjType IN ( N'VIEW' , N'SQL_TRIGGER' ) ORDER BY ur.refingObjType SELECT ur.fnObjId , ur.fnSchName , ur.fnName , ur.is_inlineable , ur.inliningStatus , COUNT(1) AS objRefCount , STRING_AGG ( CONCAT(CAST(N'' AS nvarchar(MAX)), ur.refingSchName,N'.',ur.refingObjName) , CHAR(13) + CHAR(10) ) AS aggregatedReferencingObjects FROM #UdfReferences AS ur GROUP BY ur.fnObjId , ur.fnSchName , ur.fnName , ur.is_inlineable , ur.inliningStatus ORDER BY objRefCount DESC The first result set shows the UDFs in a Table definition, Trigger or View.\nThe second result set shows how many times and which objects reference the UDF (and how hard it will be to replace across the board).\nFinding the performance stats of Scalar UDFs ; -- Previous statement must be properly terminated WITH detailPerPlan AS ( SELECT defs.database_id, defs.object_id, defs.total_worker_time, defs.execution_count, defs.total_elapsed_time, defs.total_elapsed_time / defs.execution_count AS avg_elapsed_time, defs.last_elapsed_time, defs.last_execution_time, defs.cached_time , ca.cachedSeconds , ca2.total_worker_time_s , ca2.total_elapsed_time_s , ca2.total_worker_time_s / ca.cachedSeconds AS WorkerTimeSecPerSecondsCached , ca2.total_elapsed_time_s / ca.cachedSeconds AS ElapsedTimeSecPerSecondsCached , defs.execution_count / ca.cachedSeconds AS ExecutionsPerSecondsCached FROM sys.dm_exec_function_stats AS defs WITH (NOLOCK) CROSS APPLY ( VALUES (CAST(DATEDIFF(SECOND, defs.cached_time, GETDATE()) AS decimal(15,5))) ) AS ca(cachedSeconds) CROSS APPLY ( VALUES ( defs.total_worker_time / POWER(10.,6) , defs.total_elapsed_time / POWER(10.,6) ) ) AS ca2 (total_worker_time_s, total_elapsed_time_s) ) , groupedDatabaseObject AS ( SELECT dpp.database_id , dpp.object_id , SUM(dpp.execution_count) AS execution_count_sum , SUM(dpp.total_worker_time) AS total_worker_time_sum , CAST(SUM(dpp.total_worker_time) / ((SUM(dpp.execution_count)) * 1.) AS decimal(20,2)) AS avg_worker_time_sum , SUM(dpp.total_elapsed_time) AS total_elapsed_time_sum , CAST(SUM(dpp.total_elapsed_time) / ((SUM(dpp.execution_count)) * 1.) AS decimal(20,2)) AS avg_elapsed_time_sum , CAST(SUM(dpp.total_worker_time_s) / SUM (dpp.cachedSeconds) AS decimal(20,2)) AS WorkerTimeSecPerSecondsCached_Sum , CAST(SUM(dpp.total_elapsed_time_s) / SUM (dpp.cachedSeconds) AS decimal(20,2)) AS ElapsedTimeSecPerSecondsCached_Sum , CAST(SUM(dpp.execution_count) / SUM (dpp.cachedSeconds) AS decimal(20,2)) AS ExecutionsPerSecondsCached_Sum FROM detailPerPlan AS dpp GROUP BY dpp.database_id , dpp.object_id ) SELECT CASE gdo.database_id WHEN 32767 /* https://learn.microsoft.com/en-us/sql/relational-databases/databases/resource-database?view=sql-server-ver16 */ THEN N'Resource database (Hidden)' ELSE DB_NAME (gdo.database_id) END AS [Database Name], CASE gdo.database_id WHEN 32767 THEN OBJECT_NAME (gdo.object_id) ELSE OBJECT_NAME (gdo.object_id, gdo.database_id) END AS [Function Name] , gdo.* FROM groupedDatabaseObject AS gdo ORDER BY WorkerTimeSecPerSecondsCached_Sum DESC The result could look like this. It was taken from an instance with quite a few active Scalar Functions.\nAll the values are in microseconds (unless specified otherwise). Because different plans might be in cache for various periods, the default ordering is by worker time (CPU) in seconds per second cached. But it has many dimensions, so you can order it any way you want it (if that's the way you need it).\nRecap We've covered several techniques to triage and pick Scalar UDF candidates for a rewrite. However, the call to action from the last post still applies - provide me with an example of a Scalar UDF, and I'll post a tutorial on how to rewrite it to ITVF.\nOtherwise, this post concludes the series. I hope you found it helpful.\n","permalink":"/posts/scary-scalar-functions-your-environment/","tags":["Performance","Parallelism"],"title":"Scary Scalar Functions - Part Four: Your Environment"},{"categories":["Tools"],"contents":"Foreword Not everything in the general sense, but a tool called Everything by voidtools (Download link). Usually, I have to make this distinction when googling.\nNo matter how great is my folder structure or naming conventions, there comes a time when I have trouble locating something.\nMaybe the software has a default download location which I forgot (*cough* Teams *cough*), or I want to find an install folder, or I can't locate a picture I've recently saved.\nSo what's Everything? It's a search engine for your files and folders in Windows. It's free and blazing fast.\nTaken from the FAQ:\n\"Everything\" only indexes file and folder names and generally takes a few seconds to build its database.\nA fresh install of Windows 11 (about 250,000 files) will take about 5 seconds to index.\n1,000,000 files will take about 1 minute.\nIt also keeps the indexes up to date:\nYes, \"Everything\" does monitor your file systems for all changes.\nYour search results will update in real-time to reflect any changes.\nEverything will automatically keep your NTFS indexes up to date with the NTFS USN Journal.\nChanges will not be missed when Everything is not running as the system maintains the NTFS USN Journal.\nHow to use The built-in help has easy-to-understand syntax documentation.\nThe structure of the search queries is very similar to SQL.\nYou add predicates and conditions until you filter what you want.\nMy most common search needs are limited, so I get away with a combination of a few of these strategies:\nAn allowlist of folders for a targeted search A blocklist of folders for a broad search (for example !C:\\Windows doesn't search the Windows folder) Search by extension or filetype Being a SQL Developer, I mostly search for .sql I can search for pictures or videos (multiple extensions) Search by date - e.g. I've saved the file in the past week, month, etc. Keyword somewhere in the path Once I'm happy with the results, I usually save the search in bookmarks for future use.\nThen in the result window, my most common operations are:\nLook at the preview Open Path (Ctrl+Enter) Copy Full Name to Clipboard (Ctrl+Shift+C) My use cases Working on Memes I use this filter definition when I want to find my memes for an easy upload.\nIt's a picture It has a Meme somewhere in the path (usually a parent folder) It's been created this year The search would look like this\npic: path:meme datecreated:thisyear Various folder + extension searches For example, finding all my blog posts would be searching for the Markdown files in a specific folder (\"D:\\BlogFolder\" ext:md).\nSimilarly, I have several searches for different extensions:\nPowerShell Plan explorer plans SQL scripts Visual Studio Projects etc. Content search I use this when I want to find which C# class references a Stored Procedure.\nUnfortunately, searching the content is slower, similar to SQL's leading wildcard predicate filter. Content search reads each file off disk instead of hitting the name index, so expect it to crawl compared to a normal search.\n\"D:\\GitSource\\\" file: ext:cs content:sp_WhoIsActive Finding a specific DLL When experimenting with the ScriptDom.dll, I first had to find it. It turns out lots of the 3rd party solutions use it, so I have over 15 copies on my system.\next:dll Scriptdom.dll Sql Final thoughts I've barely scratched the surface of its capabilities, and it's already convenient. It's also got glowing reviews:\nTry Everything — Shakira (Zootopia) ","permalink":"/posts/my-toolbox-everything/","tags":["Productivity"],"title":"My Toolbox - Everything"},{"categories":["Investigation"],"contents":"The problem There was a need to make changes to a table with an Indexed View. Since Indexed Views must be created with SCHEMABINDING, the View must be dropped and recreated.\nFrom past experience, I knew that this operation blocked all queries (Read/Write) that referenced any table from the View's definition for the duration of the Clustered index creation, even under the RCSI level.\nBecause the index might be large and the maintenance window small, I want to do that as fast as possible.\nResearch I’ve found several blog posts already tackling this topic in my research.\nFirst, Michael J Swart wrote about How to Create Indexed Views Online.\nMichael introduces a helper column IsMigrated that helps create an empty Indexed View instantly and then batch out the data load. The downside is that the column remains there.\nIt's an interesting approach, but since the article is seven years old (at the time of writing this blog post), I was wondering if there has been any improvement since.\nNext, Paul White (SQL Kiwi) wrote about Why does an index rebuild require a Sch-M lock?\nSome time ago, the Sch-M restriction was applied when creating an indexed view. That was pointed out to be unnecessary (Connect link no longer available) because no structure was being dropped, only created, so the behaviour was changed (to only take Sch-S and Tab-S).\nThis answer is from some three years ago.\nFrom the same year, Kendra Little wrote about Does Creating an Indexed View Require Exclusive Locks on an Underlying Table?.\nKendra's post contains a demo that proves that creating a Clustered index on the View only needs SCH-M locks on the View itself.\nBut I was still blocked The SQL Server told you to reject the evidence of your eyes and ears. — George Orwell (probably) I had all the needed proofs, but not the expected result. So I've decided to conduct the experiments myself. I'm using SQL Server 2019 Enterprise edition (the edition is essential).\nI'll reuse scripts from my previous blog post on Indexed views - IS Lock in RCSI Enabled Database to reproduce the problem.\nI'll create the Database TestLock and tables MainTable, UnrelatedTable and the view IndexedView, but let's not create the Clustered index CX_IndexedView just yet.\nWe can get all the object Ids with this query:\nSELECT o.name AS ObjectName , o.object_id AS ObjectId FROM sys.objects AS o WHERE o.is_ms_shipped = 0 AND o.type IN ('U ', 'V ') For me, the Ids are:\nObjectName ObjectId MainTable 581577110 UnrelatedTable 613577224 IndexedView 645577338 Let's open two new sessions.\nIn session 1, we'll start a transaction and the Clustered index creation. In session 2, we will read from the UnrelatedTable -- Session 1 BEGIN TRANSACTION CREATE UNIQUE CLUSTERED INDEX CX_IndexedView ON dbo.IndexedView (Id) -- Session 2 BEGIN TRANSACTION SELECT RandomColumn FROM dbo.UnrelatedTable WHERE Id \u003e (SELECT 0) /* avoiding a Trivial plan */ Session 2 should be blocked. Let's note the Ids of both sessions and plug them along with the object Ids into this script.\nSELECT DISTINCT dtl.request_session_id , dtl.request_mode , dtl.resource_associated_entity_id FROM sys.dm_tran_locks AS dtl WHERE dtl.request_session_id IN (53, 54) /* use your session Ids */ AND dtl.request_mode LIKE 'Sch-%' AND resource_associated_entity_id IN ( /* use your object Ids */ 581577110 -- MainTable , 613577224 -- UnrelatedTable , 645577338 -- IndexedView ) ORDER BY dtl.request_session_id , resource_associated_entity_id We can see that session 2 is attempting to read from the Indexed View while it's being created.\nThis is due to the feature called indexed view matching\nIf a query contains references to columns that are present both in an indexed view and base tables, and the Query Optimizer determines that using the indexed view provides the best method for executing the query, the query optimizer uses the index on the view.\nWhich is automatically attempted only in the Enterprise version (or Developer, which has the same programming surface).\nOn the one hand, usually, this helps me; on the other hand, I'm paying more money for an offline operation.\nThis behaviour is also confirmed by Paul White, who has looked under the hood:\nI confirmed the blocking happens when the query processor goes to load dependent views, anticipating that view matching might be tried later. When EXPAND VIEWS is hinted, that step is skipped, so no blocking.\nThere's no neat way to prevent automatic indexed view matching without that hint on Enterprise Edition, at least without a number of hairy side-effects.\nTo test this, we can rerun the blocked query, but this time with the hint EXPAND VIEWS.\nSELECT RandomColumn FROM dbo.UnrelatedTable WHERE Id \u003e (SELECT 0) OPTION (EXPAND VIEWS) Which is not blocked. You can rollback the transactions now and stop the blocking.\nFinally, I wasn't the only one to come across this problem.\nI'm hoping this problem can be fixed in a future CU, and we won't have to wait for a new SQL Server version.\n","permalink":"/posts/unexpected-blocking-during-the-indexed-view-creation/","tags":["Debugging","Locking","Blocking","Indexed View"],"title":"Unexpected Blocking during the Indexed View Creation"},{"categories":["Tools"],"contents":"Foreword I take a dozen screenshots on a busy day. Sharing query results, writing how-to tutorials, pointing out a mistake, or making memes. Plain Print Screen and Paint got old fast. None of the tools I've tried over the years was as good as Greenshot (download), and here's why.\nTaking the picture It's never been easier. Everyone knows to use the Print Screen button to take a screenshot of the whole screen or, along with the Alt modifier, just the current window. Greenshot takes it one step further.\nThe Capture region option comes with a handy zoom and reticle that helps with precise pixel selection. You also get the window size tooltip.\nThe Capture last region is also helpful. For example, sometimes, you take the perfect size screenshot only to notice you accidentally captured a bit of a tooltip. With this, you move the mouse and re-grab the same region.\nIt also helps capture the SSMS Grid results or Execution plans when you can rerun the query and capture the same region.\nInstructions unclear I rarely send a picture without context. Usually, it has some text, highlights, arrows, etc.\nHere's an example showcasing my favourite options.\nThe great thing about all of those is that they are objects. So after creating them, you can switch to the selection tool and move them around or transform them, unlike MS Paint, where you have to undo and try again.\nThe green bubbles with numbers in them are called counters. Very useful if you want to point out the order of operations or refer to them in text like here. Sadly you can have only one counter sequence per image. 1 Drag and drop arrow. You can choose whether you want the pointer at the start, end, both or none. 2 Text along with the typical text settings. No surprises there. 3 Underline. I wish I could make it squiggly, but alas. 4 Obfuscation. One of my most used features. You can pick a pixel size and hide sensitive data. 5 Highlight. It has several modes: Highlight text - in the screenshot. You can pick a highlight colour. Highlight area - everything but the area is out of focus. Greyscale - everything but the area is turned grey. Magnify - the selected area is zoomed over surrounding text. 6 Torn edge. You have to right-click the tool and select which edge you want to be torn. Implies there would be more data that didn't fit the screen. 7 Shapes - like you would expect in any editor. The hotkeys for these tools are intuitive and can be easily discovered from the sidebar. But don't forget to right-click each tool to find its hidden settings.\nSharing is caring Unless you want the picture idling on your computer, you'll probably want to share it. Again, Greenshot has got your back with several options.\nI usually open it in the editor and copy it to the clipboard (so I can paste it into Teams or Slack). For some edits, I still use MS Paint.\nYou can also set it up to save to a default folder so you never lose a screenshot.\nThe downside Only one comes to mind over the years of using the tool. I cannot easily combine multiple pictures.\nYou can resize the canvas, but I haven't found a cut or selection tool to move the background picture.\nI usually use MS Paint for the large canvas and copy-paste the edited pictures from Greenshot.\nI think it's just a minor inconvenience that the rest of the toolset more than makes up for.\nAnyway, I hope you've found this write-up useful, and you'll give the tool a shot.\n","permalink":"/posts/my-toolbox-greenshot/","tags":["Productivity"],"title":"My Toolbox - Greenshot"},{"categories":["Deep Dive"],"contents":" Corrections 2026-05-31: Refreshed the link to the current list of cases where inlining won't kick in and sharpened the note around it. That list was shorter when I first wrote this; it keeps growing with every cumulative update, which only confirms the warning - don't rely on inlining as your fix. Foreword In the first two parts, we have seen why the Scalar functions (UDFs) are a problem for the performance. So how do we deal with it now that we know it's a problem?\nThere is only one solution:\nI say we take off and nuke the entire site from orbit. It’s the only way to be sure. — Ellen Ripley (Aliens) That might have been hyperbole. (Or was it?) But the best solution is to eliminate UDFs as much as possible. I'll show you some ways to do that safely.\nMinimizing the impact If you're not ready to go scorched earth on the UDFs, there are some ways to optimize them.\nMake them deterministic Adding SCHEMABINDING to a Scalar UDF is necessary to make it deterministic.\nTaken from the documentation\nDeterministic functions must be schema-bound. Use the SCHEMABINDING clause when creating a deterministic function.\nAnd be sure to check this great answer by Paul White (SQL Kiwi) as well.\nDeterminism can positively impact performance and it's easily implemented as well.\nI always add SCHEMABINDING to a function that doesn't access data because it's a free win.\nOtherwise, it can be a double-edged sword. The SCHEMABINDING will unsurprisingly prevent changes to a schema, so deployments and refactorings might be more complex.\nReturn NULL on NULL input Taken from the documentation (emphasis mine)\nIf RETURNS NULL ON NULL INPUT is specified in a CLR function, it indicates that SQL Server can return NULL when any of the arguments it receives is NULL, without actually invoking the body of the function.\nThe documentation only mentions this short-circuit for CLR functions, but it works for plain TSQL scalar UDFs too. Jonathan Kehayias proved it: the same workload dropped from 31,465 calls to 13,976 (exactly the non-NULL rows), cutting CPU from 204 ms to 78 ms.\nSkipping the NULL rows means it won't be Row-by-agonizing-row (RBAR) anymore, but it will be row-by-agonizing-not-null-row (RBANNR) - which is slightly better.\nIt's hard to test all the code paths a UDF might have to see if this makes sense. But if the Scalar UDF has only one parameter, this snippet will help you construct the queries to check for the easy wins.\n; -- Previous statement must be properly terminated WITH singleParamObjects AS ( SELECT p.object_id FROM sys.parameters AS p WHERE p.name \u003c\u003e N'' GROUP BY p.object_id HAVING COUNT(1) = 1 ) SELECT o.object_id AS objId , OBJECT_SCHEMA_NAME(o.object_id) AS schName , o.name AS objName , p.name AS paramName , CONCAT ( 'SELECT ' , '''' , ca.FullName , '''' , ' AS FnName, ' , ca.FullName , '(NULL) AS FnResult;' ) AS DynamicExecute FROM sys.objects AS o JOIN sys.parameters AS p ON p.object_id = o.object_id AND p.name \u003c\u003e N'' JOIN singleParamObjects AS spp ON spp.object_id = p.object_id CROSS APPLY ( VALUES ( CONCAT ( QUOTENAME(OBJECT_SCHEMA_NAME(o.object_id)) , '.' , QUOTENAME(o.name) ) ) ) AS ca (FullName) WHERE o.type = 'FN' Constants when working with sets I sometimes see this antipattern:\nSELECT Mt.Col , dbo.CalculateValue(4) AS CalculatedValue FROM dbo.Mytable AS Mt The Scalar UDF in the SELECT clause has a constant for a parameter. A constant in a function means a constant return value. Therefore, there is no point in invoking it for every row. SQL Prompt's Code Analysis also warns about this.\nAn easy fix is to move the constant into a variable - like this:\nDECLARE @CalculatedValue int = dbo.CalculateValue(4) SELECT Mt.Col , @CalculatedValue AS CalculatedValue FROM dbo.Mytable AS Mt Removing the Scalar UDFs Scalar UDFs are a disease, a cancer of this product. It's a plague, and inlining is the cure. — Agent Smith (probably) (The Matrix) Apart from the DROP statement, the standard way to fix Scalar UDFs is inlining. Inlining is sometimes referred to as a View with parameters.\nAutomatic Scalar UDF Inlining If you are on SQL Server 2019 (Compatibility level 150), you have access to Scalar UDF Inlining - codename Froid.\nThe upside is that you don't have to change the interface.\nThe downside is that it has 20+ requirements, as documented here\nI've picked a few examples\nThe UDF doesn't invoke any intrinsic function that is either time-dependent (such as GETDATE()) or has side effects (such as NEWSEQUENTIALID()) The UDF doesn't reference table variables or table-valued parameters The UDF isn't used in a computed column or a check constraint definition The UDF doesn't contain references to Common Table Expressions (CTEs) The query invoking the UDF doesn't have Common Table Expressions (CTEs) Also, there is a list of fixed inlining bugs since its release. It's long, and together with the requirements above it means automatic inlining won't kick in for a large share of real-world UDFs. Don't lean on it as your fix: when it quietly doesn't engage, you still have the original problem and no reason to go back and rewrite anything.\nLet's repeat the UDF function in a SELECT clause from Part 1\nFirst, we need to reenable the Database scoped configuration (we turned it off in the first part to see the un-inlined behaviour) and then run the code\nUSE ScalarFunction GO ALTER DATABASE SCOPED CONFIGURATION SET TSQL_SCALAR_UDF_INLINING = ON GO SELECT TOP (10000) n.Id , n.Filler , dbo.DoNothing(n.Id) AS ScalarId FROM dbo.Nums AS n JOIN dbo.Nums AS n2 ON n.Filler = n2.Filler ORDER BY n.Id The previous plan for comparison\nThe actual execution plan XML also contains the attribute ContainsInlineScalarTsqlUdfs=\"true\".\n\u003cQueryPlan DegreeOfParallelism=\"4\" MemoryGrant=\"403720\" CachedPlanSize=\"64\" CompileTime=\"4\" CompileCPU=\"3\" CompileMemory=\"504\" ContainsInlineScalarTsqlUdfs=\"true\" \u003e Please note that every reference is evaluated separately based on context.\nIf I were to rewrite that query to use a CTE:\n; -- Previous statement must be properly terminated WITH cte AS ( SELECT n.Id , n.Filler FROM dbo.Nums AS n ) SELECT TOP (10000) n.Id , n.Filler , dbo.DoNothing(n.Id) AS ScalarId FROM cte AS n JOIN dbo.Nums AS n2 ON n.Filler = n2.Filler ORDER BY n.Id The query is not inlined because of the CTE requirement mentioned earlier. The same would apply to a CHECK constraint or a Computed column.\nYou can check which Scalar UDFs are eligible for inlining with this code.\nSELECT o.object_id AS objId , OBJECT_SCHEMA_NAME(o.object_id) AS schName , o.name AS objName , sm.is_inlineable FROM sys.sql_modules AS sm JOIN sys.objects AS o ON sm.object_id = o.object_id WHERE o.type = 'FN' AND sm.is_inlineable = 1 Manual Scalar inlining It's what it says on the box - you must rewrite it yourself to an ITVF.\nIt would be nice if the Automatic inlining had a way of providing the inlined code to make it easier, but here we are.\nITVF is a single-statement function that returns a table.\n-- Transact-SQL Inline Table-Valued Function Syntax CREATE [ OR ALTER ] FUNCTION [ schema_name. ] function_name ( [ { @parameter_name [ AS ] [ type_schema_name. ] parameter_data_type [ = default ] [ READONLY ] } [ ,...n ] ] ) RETURNS TABLE [ WITH \u003cfunction_option\u003e [ ,...n ] ] [ AS ] RETURN [ ( ] select_stmt [ ) ] [ ; ] Since we are replacing a Scalar UDF, our table should always return one row.\nOur dbo.DoNothing function rewritten as an ITVF would look like this.\nCREATE OR ALTER FUNCTION dbo.DoNothingITVF(@Id int) RETURNS TABLE WITH SCHEMABINDING AS RETURN SELECT @Id AS SameValue GO Instead of returning a data type, we are returning TABLE I add a SCHEMABINDING automatically because the function doesn't access any data. There is no BEGIN or END because it's a single-statement function The returning column has to be aliased But the work doesn't stop here. Several gotchas should be mentioned.\nManual rewrite challenges There are a few things to keep in mind before you go on a rewriting spree.\nConstraints and columns Table-valued functions cannot be used in constraints and computed columns.\nIt doesn't matter that you'll pinky swear to SQL Server that you will always return one row - it cannot be done.\nALTER TABLE dbo.ComputedColumn ADD ItvfColumn AS (SELECT TOP (1) SameValue FROM dbo.DoNothingITVF(Id)) Msg 1046, Level 15, State 1, Line 2\nSubqueries are not allowed in this context. Only scalar expressions are allowed. To remove those Scalar UDFs, you need to move the logic elsewhere. For example, Application, CRUD Procedures, Staging tables or Triggers.\nIt's not ideal, but you have to weigh the pros and cons; there is no silver bullet answer.\nDifferent functions, different calls Probably the biggest issue is that you cannot make an in-place upgrade to an ITVF.\nIf I were to try to replace the existing Scalar UDF with an ITVF:\nALTER FUNCTION dbo.DoNothing(@Id int) RETURNS table WITH SCHEMABINDING AS RETURN SELECT @Id AS SameValue GO I would get this error:\nMsg 2010, Level 16, State 1, Procedure DoNothing, Line 1\nCannot perform alter on 'dbo.DoNothing' because it is an incompatible object type. So we need to turn this:\nSELECT n.Id , dbo.DoNothing(n.Id) AS Nothing /* Scalar */ FROM dbo.Nums AS n Into this:\nSELECT n.Id , dni.SameValue AS Nothing FROM dbo.Nums AS n CROSS APPLY dbo.DoNothingITVF(n.Id) AS dni /* Table valued */ The CROSS APPLY is safe in this case because if you want to match the logic of the Scalar UDF, you must return exactly one row. Hence, never an empty result set and never multiple rows.\nWarning CROSS APPLY only matches the Scalar UDF if the ITVF returns exactly one row for every input. Return zero rows and CROSS APPLY silently drops that row; return more than one and you multiply rows. So make the ITVF always return exactly one row - I usually do that with an aggregate like MAX() and no GROUP BY, which collapses an empty result to a single NULL row and many rows down to one. To replace the invocations safely opens up a door to another challenge:\nSplit logic If you start replacing the references one by one (and usually you have to), you get into a situation when some code references the old Scalar UDF, and some are referencing the new ITVF.\nWarning While both versions coexist, someone could change the old Scalar UDF without touching the ITVF, and the two would silently drift apart. Communicate the refactoring clearly, do it promptly, and lean on code reviews. It's still a risk. Permissions Different types of functions need different permissions:\nFunction type Permission to run it Scalar UDF GRANT EXECUTE ITVF GRANT SELECT Hopefully, with Ownership chaining, it won't be a problem, but keep it in mind.\nSame result set You must always test that the new ITVF completely matches the logic of the old UDF.\nFor the quick and dirty test, I use the set operator - EXCEPT\n/* Scalar EXCEPT ITVF */ SELECT n.Id , dbo.DoNothing(n.Id) AS Nothing FROM dbo.Nums AS n EXCEPT SELECT n.Id , dni.SameValue AS Nothing FROM dbo.Nums AS n CROSS APPLY dbo.DoNothingITVF(n.Id) AS dni /* ITVF EXCEPT Scalar */ SELECT n.Id , dni.SameValue AS Nothing FROM dbo.Nums AS n CROSS APPLY dbo.DoNothingITVF(n.Id) AS dni EXCEPT SELECT n.Id , dbo.DoNothing(n.Id) AS Nothing FROM dbo.Nums AS n You have to do both directions to see a difference.\nNote EXCEPT removes duplicates and treats two NULLs as equal, so it proves the two result sets hold the same distinct rows, not that they return the same number of rows. If row multiplicity matters, compare counts as well, or lean on the automated tests mentioned below. If your UDF has more parameters, the combinations will snowball.\nI sometimes introduce temp tables (#temp) to save the results instead of running the Scalar UDF over large result sets twice.\nI use automatic tests to cover all the code paths for even more complex functions. My tool of choice is the tSQLt framework.\nCorrect data type The Data type precedence and data type inference can sometimes introduce unexpected behaviour. Therefore, one thing I always do is CAST/CONVERT the final result set to the data type of the original UDF.\nRewriting the function My example of converting dbo.DoNothing to dbo.DoNothingITVF was to illustrate the concept with an easy example. But, of course, your production functions usually won't be this simple.\nYou will have to use several tricks like CTEs or converting an empty result set to NULL with a MAX() function, etc.\nThe resulting ITVF won't be as readable, but if you are after performance, it's worth it.\nBecause a more complex example would significantly increase the scope of an already large post, I have a call to action:\nProvide me with a Minimal, Complete, and Verifiable Example of your Scalar UDF (reasonably complex), and I will pick one and show you how to inline it as the last article in the series.\nRecap In this article, we've seen how to tweak existing Scalar UDFs to minimize their impact (where possible) and the two main ways to inline them along with the challenges it brings.\nBut still, the best cure is prevention - don't write any new Scalar UDFs even if you plan to use them correctly. Someone will misuse them one day.\nIn the following article in the series, I'll share code snippets on how to find the problematic UDFs in your environment and how to prioritize them.\n","permalink":"/posts/scary-scalar-functions-the-cure/","tags":["Performance","Parallelism"],"title":"Scary Scalar Functions - Part Three: The Cure"},{"categories":["T-SQL Tuesday"],"contents":" T-SQL Tuesday #152 Hosted by Deborah Melkin Topic: What's in a name There are only two hard things in Computer Science: cache invalidation and naming things. — Phil Karlton All things should be properly named. It makes it look like there was some thought behind the design and that we don't leave the various settings to default.\nI've briefly touched on this topic in the previous T-SQL Tuesday: Coding Standards, but I cannot believe I've forgotten the one type of name I care about the most - the application name.\nWhat's my problem? Whenever I'm trying to debug a problem using sp_whoisactive or Extended Events (XE) and I see either Core Microsoft SqlClient Data Provider or .Net SqlClient Data Provider, my blood begins to boil.\nIt means I'll probably spend hours asking around to try and find the owner. Sometimes knowing the host_name helps, but there can be a multi-purpose host that runs many applications - which one is having the problem?\nThe same applies to monitoring. If I want to monitor only activity from a single app, I need something to distinguish it by. And the app_name is the perfect candidate.\nBut monitoring aside. If there is an error in your app and it takes a DBA to tell you, then you haven't been handling errors well enough.\nThe least you can do is provide the app_name as a traceback mechanism, so we can make you aware and fix the problem. Perhaps full-text search in source control or a documented overview of apps and owners (one can dream).\nAn app by any other name would perform as terribly. — Shakespeare (probably) Solution If you are not aware, go to connectionstrings.com to find some examples. Or even better, the documentation where you find all the connection string parameters.\nThe syntax is straightforward\nApplication Name=MyAppName; You can even try it in SSMS - here's an example from the blog post Scary Scalar Functions - Part Two: Performance where I want to track only XE from app_name = MonitorXE.\nYou can check that it works with SELECT APP_NAME() AS AppName.\nFinal thoughts Everything should have a name, whether it's an App, SSIS package, a script or your pet. I'll leave the convention to you because naming things is hard.\nMaybe apart from the fireplace - that one was lazy.\n","permalink":"/posts/whats-in-a-name/","tags":["Debugging","Extended Events"],"title":"What's in a name? (T-SQL Tuesday #152)"},{"categories":["Deep Dive"],"contents":" Corrections 2022-07-23: Added the Actual execution plan section thanks to feedback by Paul White (SQL Kiwi) Foreword A Scalar function that hands back its input untouched does no real work, yet it still drags a query to over 20 times slower. Part One showed how UDFs quietly kill parallelism. This time we put numbers on the damage.\nIf you want to follow along, start with the code from Part One.\nNote These numbers assume Scalar UDF Inlining is off, which the Part One setup configures (TSQL_SCALAR_UDF_INLINING = OFF). On SQL Server 2019+ (compatibility level 150) the optimizer can inline some functions and erase much of this overhead. I cover inlining, and when it kicks in, later in the series. Set up monitoring To gather the performance metrics, we'll set up additional monitoring tools. Namely:\nQuery Store (QS) Extended Events (XE) SET STATISTICS IO, TIME ON Plan Explorer If you don't have Plan Explorer, I highly recommend it.\nTo start, let's enable the QS first.\nALTER DATABASE [ScalarFunction] SET QUERY_STORE = ON (QUERY_CAPTURE_MODE = ALL) If you need to clear the QS between the runs, use this command.\nALTER DATABASE [ScalarFunction] SET QUERY_STORE CLEAR Next, we'll set up the Extended Event session.\nCREATE EVENT SESSION ScalarPerformance ON SERVER ADD EVENT sqlserver.module_end ( SET collect_statement = 1 ACTION ( sqlserver.session_id ) WHERE sqlserver.client_app_name = N'MonitorXE' AND object_type = 'FN' ) , ADD EVENT sqlserver.sql_statement_completed ( ACTION ( sqlserver.query_hash_signed , sqlserver.session_id ) WHERE sqlserver.client_app_name = N'MonitorXE' AND sqlserver.query_hash_signed \u003c\u003e 0 ) This will only track events from a client application name that equals MonitorXE. To do that, I will open a new SSMS connection and add this text to the Options\\Additional Connection Parameters tab.\nApplication Name=MonitorXE; Like so:\nYou can check that it worked with this snippet: SELECT APP_NAME() AS AppName.\nPerformance tests I'll cover several test scenarios and analyze the performance using different monitoring tools.\nThe results will be for the second executions of the queries, so we have compiled and cached plans and all pages in the buffer pool.\nDoNothing in a Select statement Queries under test SELECT TOP (10000) n.Id FROM dbo.Nums AS n ORDER BY n.Id GO SELECT TOP (10000) dbo.DoNothing(n.Id) AS Id FROM dbo.Nums AS n ORDER BY n.Id Query Store Get the performance data with this query.\nSELECT CAST(qsq.query_hash AS bigint) AS query_hash_signed , LEFT(qsqt.query_sql_text, 100) AS textSample , qsrs.last_duration , qsrs.last_cpu_time , qsrs.last_logical_io_reads , qsrs.last_rowcount , qsrs.count_executions FROM sys.query_store_query AS qsq JOIN sys.query_store_query_text AS qsqt ON qsqt.query_text_id = qsq.query_text_id JOIN sys.query_store_plan AS qsp ON qsp.query_id = qsq.query_id JOIN sys.query_store_runtime_stats AS qsrs ON qsrs.plan_id = qsp.plan_id Where qsqt.query_sql_text LIKE N'SELECT TOP%' The duration and CPU time are in microseconds.\nWe can see that the Scalar function that does nothing is over 20 times worse than its counterpart.\nExtended Events I've grouped the output by event name and aggregated the duration. The module_end event is fired for each execution of the Scalar function. That means for every row (10k). It doesn't collect the CPU time.\nThe data is very similar to Query Store's (I've collected it from separate runs). First, though, I'd like to point out some interesting info.\nThe row_count column for the query with the UDF shows double the values. I'm assuming it's for each row returned by both the query and the function. Also, the aggregated duration of the function calls is only 23 760, while the query duration is 67 134. That looks like the overhead of the UDF execution was more significant than its duration. SET STATISTICS IO, TIME ON (10000 rows affected) Table 'Nums'. Scan count 1, logical reads 275, \u003ctruncated for brevity\u003e SQL Server Execution Times: CPU time = 0 ms, elapsed time = 2 ms. (10000 rows affected) Table 'Nums'. Scan count 1, logical reads 275, \u003ctruncated for brevity\u003e SQL Server Execution Times: CPU time = 60 ms, elapsed time = 66 ms. We can see the difference here as well.\nActual execution plan The actual execution plan also shows the UDF duration in the newer schemas (starting with the sql2019 xml schema).\nShows time statistics for single query execution.\nCpuTime: CPU time in milliseconds\nElapsedTime: elapsed time in milliseconds\nUdfCpuTime: Cpu time of UDF in milliseconds\nUdfElapsedTime: Elapsed time of UDF in milliseconds\n\u003cQueryTimeStats CpuTime=\"21\" ElapsedTime=\"51\" UdfCpuTime=\"16\" UdfElapsedTime=\"16\" /\u003e Plan explorer I like how the tool highlights the UDF problem so you can see it straight away. However, one double-edged sword is that it also tries to collect the execution plans (or at least sample) of the UDFs.\nI've run into situations where my UDFs had multiple statements and ran over many rows, so the Plan Explorer collection couldn't keep up or ran out of memory.\nBut when it does, it gives you great insights, like these UDF warnings:\nI've re-run the tests for different row counts, and the differences are staggering.\nDoLookup in a Select statement I'll create a new function that actually does something for a change.\nLet's prepare some supporting tables first.\nCREATE TABLE dbo.LookupTable ( Id int NOT NULL PRIMARY KEY , LookupValue varchar(50) NOT NULL , Filler char(100) NOT NULL ) INSERT INTO dbo.LookupTable WITH (TABLOCK) (Id, LookupValue, Filler) SELECT TOP (26) n.Id, CHAR(64 + n.Id), '' FROM dbo.Nums AS n ORDER BY n.Id It's a dummy lookup table for the test.\nWhat's important is that a seek against this table uses 2 logical reads.\n(1 row affected) Table 'LookupTable'. Scan count 0, logical reads 2, \u003ctruncated for brevity\u003e SQL Server Execution Times: CPU time = 0 ms, elapsed time = 0 ms. And that's what we'll do in the Scalar function:\nCREATE OR ALTER FUNCTION dbo.DoLookup(@Id int) RETURNS varchar(50) AS BEGIN DECLARE @ReturnId varchar(50) SET @ReturnId = ( SELECT lt.LookupValue FROM dbo.LookupTable AS lt WHERE lt.Id = @Id ) RETURN @ReturnId END For the performance test, I'll use these queries.\nSELECT TOP (10000) n.Id , lt.LookupValue FROM dbo.Nums AS n LEFT JOIN dbo.LookupTable AS lt ON n.Id % 27 = lt.Id ORDER BY n.Id OPTION (MAXDOP 1) GO SELECT TOP (10000) n.Id , dbo.DoLookup(n.Id % 27) FROM dbo.Nums AS n ORDER BY n.Id OPTION (MAXDOP 1) Warning It's not a good idea to JOIN on a calculation with a modulo % operator, but I didn't want to introduce a more complex table or change the one from Part 1. It won't affect the testing. Hidden reads Let's check first with the SET STATISTICS IO, TIME ON.\n(10000 rows affected) Table 'Nums'. Scan count 1, logical reads 296, \u003ctruncated for brevity\u003e Table 'LookupTable'. Scan count 0, logical reads 20000, \u003ctruncated for brevity\u003e SQL Server Execution Times: CPU time = 10 ms, elapsed time = 11 ms. (10000 rows affected) Table 'Nums'. Scan count 1, logical reads 275, \u003ctruncated for brevity\u003e SQL Server Execution Times: CPU time = 368 ms, elapsed time = 401 ms. The first query has extra 20k reads against a LookupTable, but a duration of only 11 ms.\nThe second query shows only the 275 reads from the Nums table.\nNowhere does it show the LookupTable, and that's why the duration of 401 ms seems inexplicably high.\nWe must use the Query Store, XE or Plan Explorer to see those reads.\nQuery Store Plan Explorer The performance gap grows even further when the Scalar function repeats work for each row.\nDoNothing in a Check constraint I'll create two tables that I TRUNCATE between the test runs.\nCREATE TABLE dbo.PositiveId ( Id int PRIMARY KEY , CONSTRAINT CK_PositiveId CHECK (Id \u003e 0) ) GO CREATE TABLE dbo.PositiveIdScalar ( Id int PRIMARY KEY , CONSTRAINT CK_PositiveIdScalar CHECK (dbo.DoNothing(Id) \u003e 0) ) The test query\nINSERT INTO dbo.PositiveId (Id) SELECT TOP (10000) n.Id FROM dbo.Nums AS n ORDER BY n.Id GO INSERT INTO dbo.PositiveIdScalar (Id) SELECT TOP (10000) n.Id FROM dbo.Nums AS n ORDER BY n.Id The actual plans reveal the difference. The scalar version carries a warning on the INSERT operator and a costlier Assert:\nAnd the timings confirm it:\nDoNothing in a Computed column Unless the Computed column is persisted (not a default behaviour), there is no change in performance.\nThat's because only a definition is saved, not the actual value. Persisting a Computed column requires a deterministic function, so I'll create a new version with a slight variation.\nOtherwise, we would get this error:\nMsg 4936, Level 16, State 1, Line 428\nComputed column 'ColName' in table 'TableName' cannot be persisted because the column is non-deterministic. CREATE OR ALTER FUNCTION dbo.DoNothingDeterministic(@Id int) RETURNS int WITH SCHEMABINDING -- enables determinism AS BEGIN RETURN @Id END GO Now we can create the tables\nCREATE TABLE IdCalculation ( Id int PRIMARY KEY , IdCalculation AS Id ) GO CREATE TABLE IdCalculationScalarPersisted ( Id int PRIMARY KEY , IdCalculation AS dbo.DoNothingDeterministic(Id) PERSISTED ) And the testing queries\nINSERT INTO dbo.IdCalculation (Id) SELECT TOP (10000) n.Id FROM dbo.Nums AS n ORDER BY n.Id GO INSERT INTO dbo.IdCalculationScalarPersisted (Id) SELECT TOP (10000) n.Id FROM dbo.Nums AS n ORDER BY n.Id And finally, the performance results.\nRecap The performance of Scalar functions is horrendous even when it does nothing.\nIt worsens when the UDF has multiple statements, more complex logic, reads, etc.\nFurthermore, I consider that Scalar UDFs must be destroyed. Scalar UDF delenda est. — Tom The next article in the series will be about methods to solve the problems related to Scalar functions.\n","permalink":"/posts/scary-scalar-functions-performance/","tags":["Performance","Query Store","Extended Events"],"title":"Scary Scalar Functions - Part Two: Performance"},{"categories":["Investigation"],"contents":"The problem In this scenario, you have discovered that one of your Check constraints or Foreign keys is not trusted.\nMaybe you've detected it with a sp_Blitz, dbachecks or out of curiosity, with ad-hoc queries like these.\n/* Check constraints and their trust state */ SELECT cc.object_id AS ccId , OBJECT_NAME(cc.parent_object_id) AS tableName , cc.name AS ccName , cc.is_not_trusted FROM sys.check_constraints AS cc WHERE cc.is_disabled = 0 -- AND cc.is_not_trusted = 1 /* Foreign keys and their trust state */ SELECT fk.object_id AS fkId , OBJECT_NAME(fk.parent_object_id) AS tableName , fk.name AS fkName , fk.is_not_trusted FROM sys.foreign_keys AS fk WHERE fk.is_disabled = 0 --AND fk.is_not_trusted = 1 Run these in the context of the target database.\nWe want to fix those untrusted constraints because the optimizer won't use them when coming up with a query plan.\nRepeating offender Okay, so you went through the effort of fixing them, but the next day your constraints are not trusted again. What gives?\nIf you are sure none of the DBAs or developers is doing this to spite you, the most common culprit is a BULK INSERT or bulk copy tool (bcp).\nOne of its parameters is -h (hints)\nand one of those hints is CHECK_CONSTRAINTS\nCHECK_CONSTRAINTS\nSpecifies that all constraints on the target table or view must be checked during the bulk-import operation. Without the CHECK_CONSTRAINTS hint, any CHECK and FOREIGN KEY constraints are ignored, and after the operation the constraint on the table is marked as not-trusted.\n— Microsoft Docs (bcp utility) If you skip this parameter, bcp will mark the constraints as untrusted.\nAnd the bulk operation can be run from several different sources.\nTSQL (BULK INSERT) bcp command line .NET application SSIS No wonder you might have trouble finding the culprit.\nExtended Events to the rescue We'll set up an Extended Events (XE) session with three events to track each use case\nobject_altered - if someone really is running the ALTERs against the constraint databases_bulk_copy_rows databases_bulk_insert_rows For the bcp and bulk insert, respectively.\nWe'll grab additional audit global fields to locate the responsible process/person more quickly.\nDemo First, the XE definition.\nCREATE EVENT SESSION TrackConstraintTrust ON SERVER ADD EVENT sqlserver.databases_bulk_copy_rows (ACTION ( sqlserver.client_app_name , sqlserver.client_hostname , sqlserver.database_name , sqlserver.query_hash_signed , sqlserver.server_instance_name , sqlserver.server_principal_name , sqlserver.sql_text , sqlserver.tsql_stack ) ) , ADD EVENT sqlserver.databases_bulk_insert_rows (ACTION ( sqlserver.client_app_name , sqlserver.client_hostname , sqlserver.database_name , sqlserver.query_hash_signed , sqlserver.server_instance_name , sqlserver.server_principal_name , sqlserver.sql_text , sqlserver.tsql_stack ) ) , ADD EVENT sqlserver.object_altered (ACTION ( sqlserver.client_app_name , sqlserver.client_hostname , sqlserver.database_name , sqlserver.server_instance_name , sqlserver.server_principal_name , sqlserver.sql_text , sqlserver.tsql_stack ) WHERE ddl_phase = 'Commit' ) For the demo, Watch Live Data XE Watch Live Data Once you have an XE session running, you can stream its events live in SSMS without needing a file target. Right-click the session in Object Explorer and choose Watch Live Data.\nCaveats:\nNo memory, no history. There is no target backing the session, so you can only see events that fire while you are actively watching. Anything that happened before you opened the live view is gone. The session keeps running. Closing the Watch Live Data tab does not stop the session. It is still capturing and discarding events in the background - like a tree falling in a forest with no one around to hear it. Stop or drop the session explicitly when you are done. will do. But in your environment, you want to add an event_file target.\nStart the event session and open the Live Data window. We'll get to the output later.\nNow the environment - table and a child table with a Foreign key and Check constraint.\nCREATE DATABASE InConstraintWeTrust GO USE InConstraintWeTrust GO CREATE TABLE dbo.[Order] ( Id int IDENTITY (1,1) NOT NULL , DateCreated datetime2(3) NOT NULL CONSTRAINT DF_Order_DateCreated DEFAULT SYSDATETIME() , CONSTRAINT PK_Order PRIMARY KEY CLUSTERED (Id) ) GO CREATE TABLE dbo.OrderItem ( Id int IDENTITY (1,1) NOT NULL , OrderId int NOT NULL , ProductName varchar(50) NOT NULL , Qty int NOT NULL , CONSTRAINT PK_OrderItem PRIMARY KEY CLUSTERED (Id) , CONSTRAINT FK_OrderItem_Order_Parent FOREIGN KEY (OrderId) REFERENCES dbo.[Order] , CONSTRAINT CK_OrderItem_PositiveQty CHECK (Qty \u003e 0) ) GO INSERT INTO dbo.[Order] DEFAULT VALUES GO 10 INSERT INTO dbo.OrderItem (OrderId, ProductName, Qty) VALUES (1, 'abc', '2') , (1, 'bcd', '4') , (2, 'abc', '1') , (3, 'bcd', '2') We can check the trustworthiness state with the two queries from earlier.\nBoth constraints come back with is_not_trusted = 0, so they are trusted.\nALTER TABLE Let's test this scenario.\n/* Disable FK constraint */ ALTER TABLE dbo.OrderItem NOCHECK CONSTRAINT FK_OrderItem_Order_Parent /* Do something and enable constraint */ ALTER TABLE dbo.OrderItem CHECK CONSTRAINT FK_OrderItem_Order_Parent GO /* Disable CK constraint */ ALTER TABLE dbo.OrderItem NOCHECK CONSTRAINT CK_OrderItem_PositiveQty /* Do something and enable constraint */ ALTER TABLE dbo.OrderItem CHECK CONSTRAINT CK_OrderItem_PositiveQty After reenabling the constraints, they are not trusted.\nWe can make them trusted again with this code. Notice the WITH CHECK.\nALTER TABLE dbo.OrderItem WITH CHECK CHECK CONSTRAINT FK_OrderItem_Order_Parent ALTER TABLE dbo.OrderItem WITH CHECK CHECK CONSTRAINT CK_OrderItem_PositiveQty BULK INSERT For the other two tests, I have created a simple .csv file and saved it at this path\nD:\\OrderItems.csv\n,4,abcd,42 ,4,asdasd,42 ,5,asdasdasd,42 ,5,dhdrh,42 ,5,dasd,42 ,6,fdhg,42 I'll run the BULK INSERT statement from SSMS. The CHECK CONSTRAINTS hint is commented out on purpose.\nBULK INSERT dbo.OrderItem FROM 'D:\\OrderItems.csv' WITH ( FIRSTROW = 1, FIELDTERMINATOR = ',', --CSV field delimiter ROWTERMINATOR = '\\n', --Use to shift the control to next row TABLOCK --, CHECK_CONSTRAINTS ) Let's check the trusted status again.\n- Is the result the same?\n- Yes.\n- Did I reuse the same image?\n- Also, yes. But who cares?\nReenable the constraints WITH CHECK again for the final test.\nbcp Command line This time I'll run the utility from the Windows Terminal. Change the parameters accordingly to your DEV environment.\nbcp dbo.OrderItem in \"D:\\OrderItems.csv\" -S localhost -d InConstraintWeTrust -T -c -t ',' And let's check the trusted status for the one last time.\nTracking the culprit Let's get back to the XE output.\nWe can see all the events were successfully captured. The green highlighted events are when I restore the trusted status.\nTo track the source, we can use the client_app_name and client_hostname to narrow down the suspect.\nIf the process is nested somewhere, we can use the tsql_stack Parse TSQL Stack Parse the tsql_stack XML from an Extended Events session into a readable call stack. Paste the \u003cframes\u003e element from the XE event data into the @stackOrFrame variable.\nThe COALESCE handles both the old (handle/offsetStart/offsetEnd) and new (sqlhandle/stmtstart/stmtend) XE frame attribute names.\n/* Paste the \u003cframes\u003e\u003c/frames\u003e here */ DECLARE @stackOrFrame xml = '' ;WITH xmlShred AS ( SELECT COALESCE ( CONVERT(varbinary(64), f.n.value('.[1]/@handle', 'varchar(max)'), 1), CONVERT(varbinary(64), f.n.value('.[1]/@sqlhandle', 'varchar(max)'), 1) ) AS handle, COALESCE ( f.n.value('.[1]/@offsetStart', 'int'), f.n.value('.[1]/@stmtstart', 'int') ) AS offsetStart, COALESCE ( f.n.value('.[1]/@offsetEnd', 'int'), f.n.value('.[1]/@stmtend', 'int') ) AS offsetEnd, f.n.value('.[1]/@line', 'int') AS line, f.n.value('.[1]/@level', 'tinyint') AS stackLevel FROM @stackOrFrame.nodes('//frame') AS f(n) ) SELECT xs.stackLevel, ca.outerText, ca2.statementText FROM xmlShred AS xs CROSS APPLY sys.dm_exec_sql_text(xs.handle) AS dest CROSS APPLY (SELECT LTRIM(RTRIM(dest.text)) FOR XML PATH(''), TYPE) AS ca(outerText) CROSS APPLY ( SELECT SUBSTRING ( dest.text, (xs.offsetStart / 2) + 1, (( CASE WHEN xs.offsetEnd = -1 THEN DATALENGTH(dest.text) ELSE xs.offsetEnd END - xs.offsetStart ) / 2) + 1 ) FOR XML PATH(''), TYPE ) AS ca2(statementText) ORDER BY xs.stackLevel OPTION (RECOMPILE); I have CAST the text to XML so it's formatted nicely, but if your code contains XML-specific special characters, it might break.\nWarning The stack parsing relies on sys.dm_exec_sql_text, which reads from the plan cache. If the plan has been evicted (server restart, memory pressure, DBCC FREEPROCCACHE), the query returns NULL. Run it while the plans are still cached. and query_hash_signed columns as I blogged here:\nInvestigating Errors With Extended Events Query Hash and Query Plan Hash Mapping I hope this will help you find the runaway process and explain the situation.\n","permalink":"/posts/what-is-causing-my-constraint-to-be-untrusted/","tags":["Extended Events","Debugging"],"title":"What is causing my constraint to be untrusted?"},{"categories":["Deep Dive"],"contents":"Foreword I'm still surprised many people don't realise how lousy Scalar functions (aka UDFs) are. It's my current focus at work, and this DBA Stack Exchange question nudged me too, so I'll be revisiting this topic.\nThe focus of part one is parallelism. Unfortunately, parallelism often gets a bad rep because of the prominent wait stats. Also, if there is a skew, it can run slow. But for the most part, it's advantageous.\nWhether or not you want parallelism should be an informed choice. But Scalar functions will force the query to run serially, even if you are unaware. That's why I want to shine a light on this.\nDemo I'll be using the latest (as of the time of writing) version: Microsoft SQL Server 2022 (CTP2.0).\nAnd I'll run it in a Docker container because I'll be changing instance settings, and I don't like to affect my other tests or clean up.\nI have two reasons for using the latest version.\nFirst, you'll see that the problem is not solved yet.\nSecond, SQL 2022 shows why the query didn't go parallel (you can read about that in Erik Darling (Darling Data) wrote about non-parallel plan reasons in SQL Server 2022).\nSetting up environment Here's the Docker container if you want to follow along.\ndocker run ` -e \"ACCEPT_EULA=Y\" ` -e \"SA_PASSWORD=Password1\" ` -p 14335:1433 ` --name scalarfunction ` -d ` mcr.microsoft.com/mssql/server:2022-latest First, I'll set the instance setting Cost threshold for parallelism to a low number, so our query can run in parallel.\nEXEC sp_configure 'show advanced options', 1 RECONFIGURE EXEC sp_configure 'cost threshold for parallelism', 1 RECONFIGURE Then we can create a database. But first, I'll set the MAXDOP to 4 and disable the Scalar UDF Inlining.\nThis is a feature starting with SQL 2019 (Compatibility level 150), and it would interfere with the testing. I'll cover it in a future post in the series.\nCREATE DATABASE ScalarFunction GO USE ScalarFunction ALTER DATABASE SCOPED CONFIGURATION SET MAXDOP = 4 /* disable inlining */ ALTER DATABASE SCOPED CONFIGURATION SET TSQL_SCALAR_UDF_INLINING = OFF I'll also create a medium-sized table to have a query that consistently runs in parallel.\nDROP TABLE IF EXISTS dbo.Nums CREATE TABLE dbo.Nums ( Id int NOT NULL CONSTRAINT PK_Nums PRIMARY KEY CLUSTERED , Filler char(200) NOT NULL ) ; -- Previous statement must be properly terminated WITH L0 AS(SELECT 1 AS c UNION ALL SELECT 1) , L1 AS(SELECT 1 AS c FROM L0 CROSS JOIN L0 AS B) , L2 AS(SELECT 1 AS c FROM L1 CROSS JOIN L1 AS B) , L3 AS(SELECT 1 AS c FROM L2 CROSS JOIN L2 AS B) , L4 AS(SELECT 1 AS c FROM L3 CROSS JOIN L3 AS B) , L5 AS(SELECT 1 AS c FROM L4 CROSS JOIN L4 AS B) , Nums AS(SELECT ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) AS n FROM L5) , tally AS (SELECT TOP (5*POWER(10,5)) n FROM Nums ORDER BY n) INSERT INTO dbo.Nums WITH (TABLOCKX) (Id, Filler) SELECT n , CAST(n AS char(200)) FROM tally And the hero of the show: the parallel query that I'll use throughout the demos.\nSELECT TOP (10000) n.Id , n.Filler FROM dbo.Nums AS n JOIN dbo.Nums AS n2 ON n.Filler = n2.Filler ORDER BY n.Id Getting a nice parallel plan (notice the parallelism icon on the operator nodes)\nEnter the Scalar function I'll introduce several scenarios with a Scalar function that does absolutely nothing.\nCREATE OR ALTER FUNCTION dbo.DoNothing(@Id int) RETURNS int AS BEGIN RETURN @Id END In a SELECT clause Let's grab an Actual Execution plan for this query.\nSELECT TOP (10000) n.Id , n.Filler , dbo.DoNothing(n.Id) AS ScalarId FROM dbo.Nums AS n JOIN dbo.Nums AS n2 ON n.Filler = n2.Filler ORDER BY n.Id This time no parallelism icon on any of the operators, no Gather Streams and an extra Compute Scalar just before the SELECT.\nIf there was any doubt that the Scalar function caused this, the properties of the main node show the reason.\nNonParallelPlanReason = TSQLUserDefinedFunctionsNotParallelizable\nBut that's not all!\nI won't be posting the execution plan and non-parallel plan reason for the following demos because it's always the same.\nBut I will be showcasing some scenarios which might be surprising.\nIn a computed column CREATE TABLE dbo.ComputedColumn ( Id int PRIMARY KEY , ScalarColumn AS dbo.DoNothing(Id) ) INSERT INTO dbo.ComputedColumn (Id) VALUES (1) SELECT TOP (10000) n.Id , n.Filler FROM dbo.Nums AS n JOIN dbo.Nums AS n2 ON n.Filler = n2.Filler LEFT JOIN dbo.ComputedColumn AS cc ON 1 = 2 ORDER BY n.Id That's right. The Scalar function is referenced in a computed column of a table. Also, none of the table's columns is in the SELECT clause, and it's not even usable since I'm joining on an always false predicate 1 = 2.\nA Scalar function is so toxic that even a mere shadow of a reference prevents parallelism.\nIf you're not convinced yet, let's do another one!\nIn a column constraint CREATE TABLE dbo.CheckConstraint ( Id int PRIMARY KEY CHECK (dbo.DoNothing(Id) \u003c 10) ) INSERT INTO dbo.CheckConstraint (Id) VALUES (1), (2), (3) SELECT TOP (10000) n.Id , n.Filler FROM dbo.Nums AS n JOIN dbo.Nums AS n2 ON n.Filler = n2.Filler LEFT JOIN dbo.CheckConstraint AS cc ON 1 = 2 ORDER BY n.Id Check constraint can only reference the rows that are being inserted/updated. I usually see Scalar functions in a constraint as a misguided way to introduce a complex validation that can look at other tables and rows.\nThink about the implications! Any query that references this table will have to run serially.\nIn a View CREATE OR ALTER VIEW dbo.ViewNums AS SELECT n.Id , n.Filler , dbo.DoNothing(n.Id) AS ScalarId FROM dbo.Nums AS n GO SELECT TOP (10000) n.Id , n.Filler FROM dbo.Nums AS n JOIN dbo.Nums AS n2 ON n.Filler = n2.Filler LEFT JOIN dbo.ViewNums AS vn ON 1 = 2 ORDER BY n.Id It gives a new meaning to the \"View to kill\".\nIn a table Trigger CREATE TABLE dbo.NumsTriggered ( Id int PRIMARY KEY , Filler char(100) NOT NULL ) GO CREATE OR ALTER TRIGGER NumsTriggered_Check ON dbo.NumsTriggered AFTER INSERT AS BEGIN IF EXISTS ( SELECT i.Id FROM Inserted AS i EXCEPT SELECT dbo.DoNothing(i.Id) FROM Inserted AS i ) RETURN END Because this is a data modification, I'll run this test in a transaction that I'll roll back.\nBEGIN TRAN ; -- Previous statement must be properly terminated WITH RowsToInsert AS ( SELECT TOP (10000) n.Id , n.Filler FROM dbo.Nums AS n JOIN dbo.Nums AS n2 ON n.Filler = n2.Filler ORDER BY n.Id ) INSERT INTO dbo.NumsTriggered WITH (TABLOCKX) (Id, Filler) SELECT Id , Filler FROM RowsToInsert ROLLBACK Ok, I'll admit this one was a red herring because the Trigger itself doesn't prevent a parallel insert.\nBut the statement inside the Trigger still has the same non-parallel plan reason.\nRecap Scalar functions are evil They prevent parallelism when used in a query Even when they're not directly referenced View definition Table definition that has the Scalar function In a computed column In a check constraint Even if the view or table join condition won't evaluate to true. Note This all assumes Scalar UDF Inlining is off. On SQL 2019+ (compatibility level 150) the optimizer can inline some functions and sidestep the parallelism penalty. I'll dig into that later in the series. I will cover the performance in the next part of the series.\n","permalink":"/posts/scary-scalar-functions-parallelism/","tags":["Performance","SQL Server 2022","Parallelism"],"title":"Scary Scalar Functions - Part One: Parallelism"},{"categories":["T-SQL Tuesday"],"contents":" T-SQL Tuesday #151 Hosted by Mala Mahadevan Topic: Coding standards Corrections 2026-05-29: I say below that I \"might blog about it later.\" I did - the Scary Scalar Functions series digs into all four scalar-function problems from this list and shows the cure. I'll go with a rapid-fire strategy, just a long list of things to do and avoid with only a brief explanation. Some of these I've already covered in my rant.\nDon't do this I'll start with the don't because the risk of doing something terrible usually outweighs doing something really well.\nNo scalar functions Lots of people underestimate just how lousy scalar functions are.\nYou force the SQL to process row by agonizing row (RBAR). It's invisible to SET STATISTICS IO, TIME ON, so you might not even know how bad it is. It prevents parallelism, even if the function is buried in a column of a view you aren't even referencing. It can be hidden in Computed Columns or Check Constraints. Even though the topic is already covered by many, I might blog about it later.\nNo Merge I'll leave Aaron Bertrand wrote about why you should avoid it.\nNo magic constants WHERE p.ProductType \u003c\u003e 4 What is 4? Just set a variable (constant) from a lookup table. Or write a comment with an explanation.\nIt's the least you can do.\nNo meaningless wrappers I mean brackets and parentheses:\nWHERE ( ([p].[ProductType] \u003c\u003e 4) AND ([p].[CreateDate] \u003e= '2022-06-12 00:00:00.000') ) could just be this:\nWHERE p.ProductType \u003c\u003e 4 AND p.CreateDate \u003e= '2022-06-12 00:00:00.000' There is no need for this extra eye strain. I only use brackets when the identifier turns blue. Parentheses should be added to help readability. Also, when the order of operations might not be clear, like when AND plus OR is combined.\nNo business logic/validation in Triggers Stay away from the Triggers. I have only two use cases for them:\nScaffolding during refactoring Sync two copies of the same table INSTEAD OF Trigger on a view Enforcing audit columns (LastModificationDate, ModifiedBy) Even though now we can use Temporal tables No capes! Sorry, that's just my favourite Incredibles quote. Not sure how that got here.\nNo ordinals Apart from a quick ad-hoc query, never use this\nORDER BY 1, 3 DESC, 2 It's very brittle. Someone will come and change the select order and screw up the logic.\nIt's terrible for readability.\nNo old JOIN syntax Even though I cannot remember the last time I saw it, don't use the ANSI-89 JOIN syntax.\nSELECT * FROM dbo.Customer AS c, dbo.OrderHeader AS oh WHERE oh.CustomerId = c.CustomerId There is just no need to use this.\nIf you forget the join condition in the WHERE clause, you have a Cartesian product (row count from c × row count from oh).\nAlways* do this On the other hand, I usually recommend these (asterisk applies).\nAlias everything Especially table names. I prefer to use initials.\nNullability is specified When declaring table variables or temp tables, always specify if the column can be NULL. It helps to set the expectations about the data.\nAll constraints are named Some schema compare tools are smart enough to disregard those. But if you script out two objects and compare them with a diff tool, you will find many differences if the constraints are not explicitly named.\nScripts should be idempotent I had to search the term definition the first time around. It means that the script can be run many times but does that thing you wanted only once.\nUnless, of course, you rely on the script to break if something is amiss.\nOnly regular identifiers I'm not too fond of brackets, so don't force me to use them.\nYou don't have to type out brackets when using only regular identifiers.\nAs few hints as possible They say: \"Change is the only constant\", but it's not true. So are the hints. Once they're used, they usually stay. It's a slippery slope telling SQL how it should behave - unless you are Paul White (SQL Kiwi) .\nXACT_ABORT everywhere SQL Server is not very consistent in handling transactions, so it needs all the help it can get. Michael J Swart wrote about why you shouldn't abandon your transactions.\nAlways be deterministic If you ever wondered why the same query hands back rows in a different order from one run to the next, determinism (or the lack of it) is usually why. So I add a surrogate PK to the ORDER BY clause whenever the order has to be stable.\nThe TOP clause has parentheses A little-known fact is that the TOP operator accepts an expression. But it needs to be wrapped in parentheses like so\n[ TOP (expression) [PERCENT] [ WITH TIES ] ] Everything is schema-qualified Apart from the default schema issues, it helps differentiate real tables from table expressions.\nBut it can even lead to Compile locks blocking\nOne condition per line It makes it easier to read and move conditions around.\nDo you agree with my standards? Let me know and thank you for reading.\n","permalink":"/posts/coding-standards/","tags":["Productivity"],"title":"Coding Standards (T-SQL Tuesday #151)"},{"categories":["Deep Dive"],"contents":"SQL Server 2022 CTP is here It has been announced today (2022-05-24) during the MS Build event. The blog post includes a download link. Unfortunately, the Docker container is not quite ready yet.\nAnyway, because I'm a #TeamXE, I had to check out if there are any new goodies there. Extended Events are how I investigate errors, so a new SQL Server version is Christmas morning. So, I took an XE event list from Microsoft SQL Server 2019 (RTM-CU16) and the new one from Microsoft SQL Server 2022 (CTP2.0) and compared them.\nThe list There are precisely 500 new events between those two versions.\nAs you would expect, a big chunk of them relates to the new or improved stuff SQL 2022 is bringing.\nLike the Parameter Sensitive Plan (PSP) optimization, Cardinality Estimator Feedback and others.\nTo check the complete list yourself, you can import it into a temp table from this gist - New Extended Events in SQL Server 2022.\nInteresting events After I read the list, a few events caught my attention.\nquery_abort Indicates that a cancel operation, client-interrupt request, or broken client connection was received, triggering abort.\nI can replace my old rpc_completed where result = 2 (abort) trick with this event instead.\nIt also shows the KILL statements from other sessions.\ntsql_feature_usage_tracking Track usage of t-sql features in queries for the database\nThe event returns a column features_used - Provides a bitmap of features used in query.\nI couldn't get this event to fire, and even if I did, I probably couldn't parse the bitmap without documentation.\nquery_antipattern Occurs when a a query antipattern is present and can potentially affect performance.\n(The quote is verbatim, so I left the typo in there).\nSo far it only identifies these antipatterns:\nTypeConvertPreventingSeek NonOptimalOrLogic LargeIn LargeNumberOfOrInPredicate Max But I think it has potential.\nAnyway, which event do you think will be useful? Let me know!\n","permalink":"/posts/new-extended-events-in-sql-server-2022/","tags":["SQL Server 2022","Extended Events"],"title":"New Extended Events in SQL Server 2022"},{"categories":["How to"],"contents":" Corrections 2026-05-29: Fixed an off-by-one in the greedy-path loop. It ran one iteration too many and inserted a redundant final row (silently dropped by the join). The loop now stops after the last cheat. Foreword I've started to play Lego Video games with my daughter, and since it's one of her first games, it can be a little frustrating.\nEven though there are no high stakes (you can't lose in those games), I thought maybe playing with the cheats might make a better experience.\nIf you are not familiar with the games, you enter cheats via an interface like this:\nDespite playing on a computer and with a perfectly usable keyboard, I cannot type the input directly.\nInstead, I must use this six-position combination lock that uses letters and numbers.\nIt's cyclic and goes from A to Z, 0 to 9 and back to A again.\nUp goes forwards; Down goes backwards.\nI thought about finding an efficient way of input because it’s not enough to enter and unlock the cheat once.\nYou have to reenter the cheat for every new game session.\nThe plan I'll create a table representing one dial with letters and numbers in sequence. Then I'll generate all possible combinations and calculate the shortest distance. Is it faster to change from M to 5 going up or down? Now that I can calculate the distance for letters, I'll create a function that does the same but for the whole cheat codes. I'll generate all code combinations. Finally, I will use a greedy algorithm to find a short path. The code Create the LetterSequence First, we'll create the database and table to hold our sequence.\nCREATE DATABASE LegoCheats GO USE LegoCheats CREATE TABLE dbo.LetterSequence ( SequenceNo tinyint IDENTITY NOT NULL INDEX IX_SequenceNo UNIQUE , Letter char(1) NOT NULL CONSTRAINT PK_LetterOrder PRIMARY KEY ) I wish I had the STRING_SPLIT parameter enable_ordinal, but at the time of writing, it's available only on Azure SQL and is planned for SQL Server 2022.\nNote enable_ordinal shipped in SQL Server 2022, so on a current instance you no longer need the IDENTITY trick below. So I'll have to use the target table Identity to simulate that.\nTo get the string, I wrote abcdefghijklmnopqrstuvwxyz0123456789 and then used RegEx to add a separator.\nFind \\B, Replace ,\nINSERT INTO dbo.LetterSequence (Letter) SELECT UPPER(value) FROM STRING_SPLIT('a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z,0,1,2,3,4,5,6,7,8,9', ',') Create the LetterDistance Now we calculate the shortest distance for each combination.\nAfter creating the table, I'll grab the maximum sequence number (36).\nWhen I cross the boundary between the start and end of the sequence, I'll add the @maxLetterSeqno and use the modulo operator (%) to wrap around.\nWhen the distance is the same both ways (like 0 or 18), I'll pick the direction as DOWN because it doesn't matter.\nCREATE TABLE dbo.LetterDistance ( LetterFrom char(1) NOT NULL , LetterTo char(1) NOT NULL , Distance tinyint NOT NULL , Direction varchar(10) NULL , CONSTRAINT PK_LetterDistance PRIMARY KEY (LetterFrom, LetterTo) ) DECLARE @maxLetterSeqno tinyint = (SELECT MAX(SequenceNo) FROM dbo.LetterSequence) INSERT INTO dbo.LetterDistance (LetterFrom, LetterTo, Distance, Direction) SELECT f.Letter AS FromLetter , t.Letter AS ToLetter , md.minDist , md.direction FROM dbo.LetterSequence AS f CROSS JOIN dbo.LetterSequence AS t CROSS APPLY ( VALUES ( CASE WHEN f.SequenceNo \u003e= t.SequenceNo THEN f.SequenceNo - t.SequenceNo ELSE ((@maxLetterSeqno + f.SequenceNo - t.SequenceNo) % @maxLetterSeqno) END, CASE WHEN t.SequenceNo \u003e= f.SequenceNo THEN t.SequenceNo - f.SequenceNo ELSE ((@maxLetterSeqno + t.SequenceNo - f.SequenceNo) % @maxLetterSeqno) END ) ) AS distance (MoveMinus, MovePlus) CROSS APPLY ( VALUES ( CASE WHEN distance.MoveMinus \u003c= distance.MovePlus THEN distance.MoveMinus ELSE distance.MovePlus END , CASE WHEN distance.MoveMinus \u003c= distance.MovePlus THEN 'DOWN' ELSE 'UP' END ) ) AS md(minDist, direction) Here's the answer to the earlier question (shortest path from M to 5)\nSELECT * FROM dbo.LetterDistance AS ld WHERE ld.LetterFrom = 'M' AND ld.LetterTo = '5' Calculate word distance Now that I can calculate the individual letter distance, I'll move on to the whole words.\nI'll create an ITVF that takes two words as input and returns the total distance plus instructions. I'm not counting the movement required to switch between the dials.\nCREATE OR ALTER FUNCTION dbo.WordDistance ( @wordFrom char(6) , @wordTo char(6) ) RETURNS table AS RETURN WITH unpivotLetters AS ( SELECT SUBSTRING(@wordFrom, cj.Position, 1) AS letterFrom , SUBSTRING(@wordTo, cj.Position, 1) AS letterTo , cj.Position FROM (VALUES (1), (2), (3), (4), (5), (6)) AS cj(Position) ) SELECT SUM(ca.Distance) AS WordDistance , MAX(IIF(ul.Position = 1, CONCAT_WS(' ', ca.Distance, ca.Direction), NULL)) AS Letter1 , MAX(IIF(ul.Position = 2, CONCAT_WS(' ', ca.Distance, ca.Direction), NULL)) AS Letter2 , MAX(IIF(ul.Position = 3, CONCAT_WS(' ', ca.Distance, ca.Direction), NULL)) AS Letter3 , MAX(IIF(ul.Position = 4, CONCAT_WS(' ', ca.Distance, ca.Direction), NULL)) AS Letter4 , MAX(IIF(ul.Position = 5, CONCAT_WS(' ', ca.Distance, ca.Direction), NULL)) AS Letter5 , MAX(IIF(ul.Position = 6, CONCAT_WS(' ', ca.Distance, ca.Direction), NULL)) AS Letter6 FROM unpivotLetters AS ul CROSS APPLY ( SELECT ld.Distance, ld.Direction FROM dbo.LetterDistance AS ld WHERE ld.LetterFrom = ul.letterFrom AND ld.LetterTo = ul.LetterTo ) AS ca Score word combinations To test this out, I have prepared a list of cheats for the Lego Batman: The Videogame that I want to apply.\nI will generate all combinations (except with itself), calculate the word distance and save them into a WordCombination table.\n/* Prepare the tables */ CREATE TABLE dbo.WordCombination ( IdFrom tinyint NOT NULL , WordFrom char(6) NOT NULL , WordFromDescription varchar(70) NULL , IdTo tinyint NOT NULL , WordTo char(6) NOT NULL , WordToDescription varchar(70) NULL , TotalDistance int NOT NULL , CONSTRAINT PK_WordCombination PRIMARY KEY (IdFrom, IdTo) ) CREATE TABLE #CheatCodes ( Id tinyint NOT NULL , Word char(6) NOT NULL , CheatDescription varchar(70) NULL ) /* Insert the cheat codes, I include the starting point as well */ INSERT INTO #CheatCodes (Id, Word, CheatDescription) VALUES (01, 'AAAAAA', 'Initial position') ,(02, 'ML3KHP', 'Extra Hearts') ,(03, 'JRBDCB', 'Faster Batarangs') ,(04, 'EVG26J', 'Faster Piece Assembly') ,(05, 'ZOLM6N', 'Faster Walking') ,(06, 'D8NYWH', 'Flaming Batarangs') ,(07, 'XPN4NG', 'Frozen Batarangs') ,(08, 'HJH7HJ', 'Heart Regeneration') ,(09, 'JXUDY6', 'Immunity to Freeze') ,(10, 'WYD5CP', 'Invincibility') ,(11, 'ZXGH9J', 'Minikit Detector') ,(12, '9LRGNB', 'Multiply Score') ,(13, 'KHJ544', 'Piece Detector') ,(14, 'MMN786', 'Power Brick Detector') ,(15, 'N4NR3E', 'Score Multiplier x2') ,(16, 'CX9MAT', 'Score Multiplier x4') ,(17, 'MLVNF2', 'Score Multiplier x6') ,(18, 'WCCDB9', 'Score Multiplier x8') ,(19, '18HW07', 'Score Multiplier x10') INSERT INTO dbo.WordCombination ( IdFrom , WordFrom , WordFromDescription , IdTo , WordTo , WordToDescription , TotalDistance ) SELECT f.Id , f.Word , f.CheatDescription , t.Id , t.Word , t.CheatDescription , wd.WordDistance FROM #CheatCodes AS f /* from */ CROSS JOIN #CheatCodes AS t /* to */ CROSS APPLY dbo.WordDistance(f.Word, t.Word) AS wd WHERE f.Id \u003c\u003e t.Id /* all combinations except itself */ /* Check the result */ SELECT * FROM dbo.WordCombination AS wc ORDER BY wc.IdFrom, wc.TotalDistance Finding an efficient path Ideally, I want to enter every cheat from my #CheatCodes temp table with as few moves as possible. That's 18 entries (the first row is the initial state).\nI've chosen to use a greedy algorithm that uses a loop, for simplicity.\nI will start on the AAAAAA position and pick the first word with the lowest distance until all words have been visited.\nCREATE TABLE #FinalPath ( SeqNo tinyint IDENTITY NOT NULL , IdFrom tinyint NOT NULL , IdTo tinyint NOT NULL ) DECLARE @LoopCounter tinyint = 1 DECLARE @IdFrom tinyint = 1 /* AAAAAA */ DECLARE @IdTo tinyint DECLARE @maxId tinyint = (SELECT MAX(cc.Id) FROM #CheatCodes AS cc) WHILE @LoopCounter \u003c @maxId BEGIN SELECT TOP (1) @IdTo = wc.IdTo FROM dbo.WordCombination AS wc WHERE wc.IdFrom = @IdFrom AND NOT EXISTS ( SELECT * FROM #FinalPath AS fp WHERE fp.IdFrom = wc.IdTo ) ORDER BY wc.TotalDistance INSERT INTO #FinalPath (IdFrom, IdTo) SELECT @IdFrom, @IdTo SET @IdFrom = @IdTo SET @LoopCounter += 1 END SELECT wc.WordFrom , wc.WordTo , wc.WordToDescription , SUM(wd.WordDistance) OVER () AS TotalDistanceSum , wd.WordDistance , wd.Letter1 , wd.Letter2 , wd.Letter3 , wd.Letter4 , wd.Letter5 , wd.Letter6 FROM #FinalPath AS fp JOIN dbo.WordCombination AS wc ON fp.IdFrom = wc.IdFrom AND fp.IdTo = wc.IdTo CROSS APPLY dbo.WordDistance(wc.WordFrom, wc.WordTo) AS wd ORDER BY fp.SeqNo The full run lands at a total distance of 638 across all 18 codes.\nI have to test it on the real thing, of course.\nOptimal path I've used the word \"efficient\" instead of \"optimal\" on purpose. That's because this is a famous computer science problem called Travelling salesman problem.\nGiven a list of cities and the distances between each pair of cities, what is the shortest possible route that visits each city exactly once and returns to the origin city? — (Wikipedia) And finding the optimal path would significantly increase the scope of this blog post. There is always a brute force solution, but the number of combinations ramps up quickly.\nI've tried brute-forcing the solution for the first 12 rows (1 starting position + 11 cheats), and it took my PC 1 hour and 6 minutes to come up with the best path 1-3-9-12-7-10-11-5-2-8-4-6 for the total distance of 409.\nThe greedy algorithm came up with 1-3-8-4-11-5-10-7-12-9-6-2 (total distance = 456), which is good enough for me.\nIt is an interesting problem, so I might revisit it in a future blog post.\n","permalink":"/posts/efficient-cheating-at-lego-video-games/","tags":["Regex"],"title":"Efficient Cheating at Lego Video Games"},{"categories":["How to"],"contents":"Foreword A stored procedure in one database needs to read a table in another, and you get slapped with:\nThe server principal \"Timmy\" is not able to access the database \"TargetDB\" under the current security context. You could open up cross-database ownership chaining and call it a day, but that's a security hole waiting to happen. Module signing is the secure way - and it has just enough moving parts that I forget the details every time I leave it alone for a few months.\nSo I'm writing this one mainly as a reminder for myself. The most helpful parts will be the diagram detailing all the components and how they relate, plus a comprehensive example anyone can follow.\nI'm not going to cover Module Signing in general (I'll leave that to Solomon Rutzky ).\nNor will I cover other ways to achieve Cross DB access (like Cross DB Ownership chaining) because this is superior from the security standpoint.\nComponents A picture is worth a thousand words.\nBut commentary also helps. We have two databases, SourceDB and TargetDB.\nSourceDB contains a module (e.g. Stored Procedure) that wants to access some table from TargetDB.\nTo do that, we're going to need:\nTwo databases A table in the TargetDB A stored procedure in the SourceDB Certificate in each database The Certificate in SourceDB must have a Private key The Certificate in the TargetDB must have the same thumbprint as the one in SourceDB If it's only one-way communication, it doesn't need the Private key A user created from Certificate in the TargetDB Only because we cannot grant permissions directly to a certificate Add the signature to the stored procedure in SourceDB Demo time Because we'll be switching the context of the two databases fairly often, I'll start each code block with the USE DbName of the specific context.\nIt might be easier to follow along if you have two side by side sessions - one for each database.\nFirst, we create the environment\nUSE [master] CREATE DATABASE SourceDB CREATE DATABASE TargetDB Then let's create a table in each DB and fill it with some data. I've chosen the table names to have the same initial as their respective DB.\nUSE TargetDB CREATE TABLE dbo.Tony ( Id int PRIMARY KEY , Letter char(1) ) INSERT INTO dbo.Tony (Id, Letter) VALUES (1, 'T') , (2, 'O') , (3, 'N') , (4, 'Y') /* change context */ USE SourceDB CREATE TABLE dbo.Stark ( Id int PRIMARY KEY , Letter char(1) ) INSERT INTO dbo.Stark (Id, Letter) VALUES (5, 'S') , (6, 'T') , (7, 'A') , (8, 'R') , (9, 'K') Back in the SourceDB we'll create the self-signed certificate. The password has to conform to the Password complexity rules.\nNote Simplified version:\nThe password is at least eight characters long Contains at least 3 out of these 4 categories Uppercase letter Lowercase letter Number Special character USE SourceDB CREATE CERTIFICATE CrossDb_Cert ENCRYPTION BY PASSWORD = 'Cert_Private_Key' WITH SUBJECT = 'Used for Cross DB access' , EXPIRY_DATE = '99991231' Warning The passwords in this walkthrough (Cert_Private_Key, Password1, and so on) are deliberately obvious so the demo is easy to follow. Use proper secrets in any real environment. We can check the existence with this snippet\nUSE SourceDB SELECT c.name , c.pvt_key_encryption_type_desc , c.thumbprint , c.sid FROM sys.certificates AS c This is my result\nNow we have to export the certificate from the SourceDB and import it into TargetDB. Because I like to source control my script, I prefer TSQL code instead of backing up the certificate to file.\nFor one way only, the public portion of the certificate will be enough.\nLet's get the binary value\nUSE SourceDB SELECT CERTENCODED(CERT_ID('CrossDb_Cert')) AS PublicPortionBinary And copy-paste it into this snippet on the TargetDB.\nWe'll go ahead and create the User from the Certificate and grant them SELECT permissions in one fell swoop.\nUSE TargetDB CREATE CERTIFICATE CrossDb_Cert FROM BINARY = /* \u003c-- Paste the public binary value here (e.g. 0x308202D0…) */ CREATE USER CrossDb_CertUser FROM CERTIFICATE CrossDb_Cert GRANT SELECT ON dbo.Tony TO CrossDb_CertUser We can check the creation of the cert and user with this snippet:\nUSE TargetDB SELECT c.name AS certName , c.pvt_key_encryption_type_desc , c.sid AS certSid , c.thumbprint , dp.name AS UserFromCert FROM sys.certificates AS c JOIN sys.database_principals AS dp ON c.sid = dp.sid We can also check that the certificate has the same thumbprint\nSELECT c.name, c.thumbprint FROM SourceDB.sys.certificates AS c INTERSECT SELECT c.name, c.thumbprint FROM TargetDB.sys.certificates AS c The only thing remaining is the stored procedure.\nUSE SourceDB GO CREATE OR ALTER PROCEDURE dbo.ReadFromTargetDB AS BEGIN SELECT * FROM TargetDB.dbo.Tony END Testing Typically, I would create a User without login and impersonate it, but impersonating just the User wouldn't work - you would have to impersonate the Login.\nSo we will do this the old fashioned way - create Login, User from Login and grant it permissions to execute the stored procedure.\nCREATE LOGIN Timmy WITH PASSWORD = 'Password1' GO USE SourceDB CREATE USER Timmy FROM LOGIN Timmy GRANT EXECUTE ON dbo.ReadFromTargetDB TO Timmy Now we have to open a new session and use SQL Server Authentication to log in as a Timmy.\nAnd run the procedure\nUSE SourceDB EXEC dbo.ReadFromTargetDB This returns an error because we have not yet signed the procedure\nMsg 916, Level 14, State 2, Procedure dbo.ReadFromTargetDB\nThe server principal \"Timmy\" is not able to access the database \"TargetDB\" under the current security context. Let's sign it then. Switch to the admin session (not the Timmy session) and run\nUSE SourceDB ADD SIGNATURE TO dbo.ReadFromTargetDB BY CERTIFICATE CrossDb_Cert WITH PASSWORD = 'Cert_Private_Key' -- the Private key from the self-signed certificate If it passes, you have confirmation that the password was correct.\nOtherwise, you would see this error\nMsg 15466, Level 16, State 9\nAn error occurred during decryption. You can also check with this snippet:\nUSE SourceDB SELECT c.name AS certName , OBJECT_NAME(cp.major_id) AS objectName , cp.crypt_type_desc FROM sys.certificates AS c LEFT JOIN sys.crypt_properties AS cp ON cp.thumbprint = c.thumbprint Switch back to the Timmy session and rerun the procedure, which now executes without errors.\nImpersonation I once spent a long time debugging an issue when there wasn't one.\nI was just too lazy to create the Login along with the test user and tried to cut corners. (This is the same trap I warned about in Short Code Examples - you can't use plain impersonation to test module signing permissions.)\nTo refresh the terminology:\nLogin = Server scoped principal User = Database scoped principal In this test, I'll impersonate the existing Timmy Login and try to run the procedure.\nThen I'll create a new user without Login, impersonate it and attempt it again.\nUSE SourceDB EXECUTE AS LOGIN = 'Timmy' /* Impersonating a Server principal */ /* After impersonation */ SELECT SUSER_NAME() AS serverLogin , USER_NAME() AS dbUser , ORIGINAL_LOGIN() AS originalLogin EXEC dbo.ReadFromTargetDB REVERT -- end the impersonation Tip After impersonation, always check that the SUSER_NAME() equals ORIGINAL_LOGIN(). If not, run the REVERT again or open a new session. This works. Now for the user without login:\nUSE SourceDB CREATE USER NewTimmy WITHOUT LOGIN GRANT EXECUTE ON dbo.ReadFromTargetDB TO NewTimmy EXECUTE AS USER = 'NewTimmy' /* Impersonating a Database principal */ /* After impersonation */ SELECT SUSER_NAME() AS serverLogin , USER_NAME() AS dbUser , ORIGINAL_LOGIN() AS originalLogin EXEC dbo.ReadFromTargetDB REVERT -- end the impersonation Msg 916, Level 14, State 2, Procedure dbo.ReadFromTargetDB\nThe server principal \"S-1-9-3-1136980307-1270705754-1660177281-2973733659\" is not able to access the database \"TargetDB\" under the current security context. Two-way access Increase speed, drop down and reverse direction! — Lrrr (Futurama) So far, we have been executing the procedure from SourceDB and reading from TargetDB. It won't work the other way around with the current form of the certificate.\nUSE TargetDB GO CREATE OR ALTER PROCEDURE dbo.ReadFromSourceDB AS BEGIN SELECT * FROM SourceDB.dbo.Stark END GO ADD SIGNATURE TO dbo.ReadFromSourceDB BY CERTIFICATE CrossDb_Cert WITH PASSWORD = 'Cert_Private_Key' Msg 15556, Level 16, State 1\nCannot decrypt or encrypt using the specified certificate, either because it has no private key or because the password provided for the private key is incorrect. That's because the certificate on the TargetDB doesn't have a private key\nUSE TargetDB SELECT c.name , c.pvt_key_encryption_type_desc , c.thumbprint , c.sid FROM sys.certificates AS c To fix this, we'll do the following steps:\nDelete the user created from the certificate Delete the certificate Copy both the public and the private keys from the SourceDB cert Create a new certificate with a private key in the TargetDB Create a user in the SourceDB to grant permissions on the Stark table Sign the ReadFromSourceDB procedure with the certificate Steps 1 and 2 are easy\nUSE TargetDB DROP USER CrossDb_CertUser DROP CERTIFICATE CrossDb_Cert To script out the certificate completely, we have to know the original password (Cert_Private_Key).\nUSE SourceDB SELECT CERTENCODED(CERT_ID('CrossDb_Cert')) AS PublicPortionBinary SELECT CERTPRIVATEKEY ( CERT_ID('CrossDb_Cert') , 'CustomEncryptPwd1' /* Encryption password */ , 'Cert_Private_Key' /* Private key */ ) AS PrivateKeyBinary The encryption password can be anything conforming to the complexity rules. It will be only used once in the next code snippet. If you didn't use the correct password, the function would return NULL instead of the binary string.\nCopy those two values and paste them here to create the cert in TargetDB (now with private key)\nUSE TargetDB CREATE CERTIFICATE CrossDb_Cert FROM BINARY = /* \u003c-- Paste the public binary value here (e.g. 0x308202D0…) */ WITH PRIVATE KEY ( BINARY = /* \u003c-- Paste the private binary value here (e.g. 0x1EF1B5B0…) */ , DECRYPTION BY PASSWORD = 'CustomEncryptPwd1' /* Decrypt using the same password from the previous step */ , ENCRYPTION BY PASSWORD = 'NewPrivateKeyPwd1' /* You can set the same or new private key password */ ) To showcase something, I've changed the original private key from Cert_Private_Key to NewPrivateKeyPwd1.\nThis is useful if you deploy to different environments and want a known private key for the local environment but a secret private key for the production.\nLet's speedrun the rest of the steps.\nDon't forget to use the new password when signing the procedure in the TargetDB.\nUSE SourceDB /* Create user from certificate and grant select permission on the table */ CREATE USER CrossDb_CertUser FROM CERTIFICATE CrossDb_Cert GRANT SELECT ON dbo.Stark TO CrossDb_CertUser USE TargetDB /* Sign the procedure */ ADD SIGNATURE TO dbo.ReadFromSourceDB BY CERTIFICATE CrossDb_Cert WITH PASSWORD = 'NewPrivateKeyPwd1' /* new private key in the TargetDB */ /* Create another testing login and user, grant permission on the procedure */ CREATE LOGIN AnotherTimmy WITH PASSWORD = 'Password1' CREATE USER AnotherTimmy FROM LOGIN AnotherTimmy GRANT EXECUTE ON dbo.ReadFromSourceDB TO AnotherTimmy /* Impersonate the new login and exec the procedure */ EXECUTE AS LOGIN = 'AnotherTimmy' EXECUTE dbo.ReadFromSourceDB And AnotherTimmy reads the Stark table across databases without any ownership chaining. Traffic now flows both ways.\nConclusion Module signing has a lot of pieces, but they always fit together the same way:\nA certificate with a private key lives in the database that holds the module (the signer). The same certificate, public portion only, lives in the database that holds the data, with a user created from it and granted the permissions. The module gets signed, so callers borrow the certificate's permissions only while that module runs - no broad rights handed out, no ownership chaining switched on. For two-way access, both certificates need the private key so each side can sign its own module.\nThat's it, folks! I hope my future self will thank me for all the unnecessary details.\n","permalink":"/posts/cross-db-access-with-module-signing/","tags":["Security","Debugging"],"title":"SQL Server Module Signing for Secure Cross-Database Access"},{"categories":["Opinion"],"contents":" Warning It's hard to convey sarcasm in text. I'm putting this disclaimer at the top so I don't come off as egotistical. This post is meant as hyperbole, and I'll leave finding the truth of any statements to the reader. I'm allergic to semicolons Aaron Bertrand wrote about Why I always start CTEs with a statement terminator.\nSo I'll take it one step further.\nI'll never use semicolons unless I have to.\nTools like Redgate's SQL Prompt can add semicolons automatically, but I still won't do it.\nThese are the reasons why\nThey don't do anything for me. I can see where the statement ends. If I can't, then a semicolon wouldn't help anyway They are clearly not required by the engine either Requiring semicolons is not enforceable. It was deprecated in 2008 R2 (in 2010) - but here we are. Features Not Supported in a Future Version of SQL Server\nNot ending Transact-SQL statements with a semicolon.\nEnd Transact-SQL statements with a semicolon ( ; ).\nPractical reasons My code is not set in stone. So I have to move this clinging semicolon out of the way whenever I want to update.\nSelecting the space before the semicolon is like separating two thin Lego pieces. I hope you have a large font and precise mouse\nLet's take this query as an example\nSELECT dopc.instance_name , dopc.cntr_value FROM sys.dm_os_performance_counters AS dopc WHERE dopc.object_name = N'SQLServer:Deprecated Features' AND dopc.cntr_value \u003e 0; -- I bet you hate this comment now Removing the line Without the semicolon With the semicolon Select the line Select the line Shift+Del Shift+Del Up Arrow End ; Adding new lines Without the semicolon With the semicolon Select the empty line below Pixelhunt the space before the semicolon first (or say goodbye to IntelliSense) Add the condition Delete the ; Enter Select the empty line below Add the condition ; Enter It's just more work for no benefit at all.\nHeck, I don't even end all my messages with punctuation.\nAnd be honest with yourself, do you?\nI've skipped a few full stops on purpose, have you noticed?\nThe controversy doesn't end here.\nThe leading commas First, let me ask you this. Do you write the conditions in the WHERE clause like this\nWHERE o.is_ms_shipped = 1 AND parent_object_id IS NULL AND o.name LIKE 'tbl%' Or like this?\nWHERE o.is_ms_shipped = 1 AND parent_object_id IS NULL AND o.name LIKE 'tbl%' I'm using the first style because\nI like knowing where the new condition begins It's easier to add lines or move them around The logic starts in the same column Me using leading commas is just being consistent.\nPractical reasons Having all commas moved out of the way at the start of the line is good for mass column updates by holding Alt and dragging the mouse. Also, since the whitespace at the end doesn't count, I can quickly wrap my columns in code and add aliasing, comments, etc.\nNot saying converting all columns to VARCHAR(200) is a good idea, but here is a demo:\nMass-editing every column to VARCHAR(200) in one stroke with Alt+Drag block selection in SSMS. I'm mainly using SSMS. The Visual Studio Code or Azure Data Studio are close with the Multi-cursor (drag middle button), but not quite.\nAliasing Just when you thought you couldn't hate my style more - BAM, out of the left-field with the aliasing.\nI like using initials as an alias\nIt's short, so it helps with the readability I can usually deduce the full table name You should qualify your columns, and there are tables like this out there tblCustomerContactFormApproval - Imagine qualifying your columns with this The usual counterargument is (surprisingly) also readability.\n\"How can you tell what table p means? It's better to use the full table name or meaningful alias\"\nWell, that's easy, you see.\nI start reading the code in the FROM clause. Also, I've trained myself to keep multiple things in short-term memory.\nThis handy script shows you the number of conflicting alias initials.\n/* The Clash of Aliases */ ;WITH L0 AS(SELECT 1 AS c UNION ALL SELECT 1), L1 AS(SELECT 1 AS c FROM L0 CROSS JOIN L0 AS B), L2 AS(SELECT 1 AS c FROM L1 CROSS JOIN L1 AS B), L3 AS(SELECT 1 AS c FROM L2 CROSS JOIN L2 AS B), Nums AS(SELECT ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) AS n FROM L3) , tally AS (SELECT TOP (256) n FROM Nums ORDER BY n) , aliases AS ( SELECT t.name AS tabName , STRING_AGG(ca.chr, '') WITHIN GROUP (ORDER BY ca.seqNo) AS abb FROM sys.tables t CROSS APPLY( SELECT SUBSTRING(t.name, n, 1), n from tally n) ca(chr, seqNo) WHERE PATINDEX('[A-Z]', ca.chr) \u003e 0 AND ca.chr = UPPER(ca.chr) COLLATE Latin1_General_CS_AI GROUP BY t.name ) SELECT a.tabName , LOWER(a.abb) AS abb , COUNT(1) OVER (PARTITION BY a.abb) AS cnt FROM aliases a ORDER BY cnt DESC, a.tabName What about the other stuff? I haven't covered\nIndentation and Tabs vs Spaces UPPERCASE, lowercase, camelCase, etc. Extra words like INNER JOIN vs JOIN And that's because most of the time, I don't care. I only care if it has any practical implications (performance, readability, etc.), but none of it is a hill I'm willing to die on. Not even the semicolons or leading commas are worth it.\nTry to make an argument for your code style, but in the end, agreed guidelines and consistency should always win. Just hope that there is an automatic formatter that can fix your code.\n","permalink":"/posts/why-my-way-of-writing-sql-is-superior/","tags":["Productivity"],"title":"Why my way of writing SQL is superior"},{"categories":["How to"],"contents":"The problem You pull a query_hash from a Dynamic Management Object (DMO) or Query Store and get a tidy binary(8). You pull the same hash from Extended Events and get a big, often negative, number. Same query, same hash, two representations that stubbornly refuse to match.\nI work with the Query Hash and Query Plan Hash (I'll refer to them collectively as the Hashes) all the time across DMOs, Query Store and Extended Events. So let's cover the efficient ways of mapping between those types.\nI'm expecting a certain familiarity with these concepts. If you are not familiar, I recommend Bart Duncan 's article Query Fingerprints and Plan Fingerprints.\nData types DMOs These are all the DMOs that reference the Hashes.\nSELECT sc.object_id , so.name AS objName , so.type_desc , sc.name AS columnName , t.name AS typeName , sc.max_length FROM sys.system_columns AS sc JOIN sys.system_objects AS so ON so.object_id = sc.object_id JOIN sys.types AS t ON t.system_type_id = sc.system_type_id AND t.user_type_id = sc.user_type_id WHERE sc.name IN ('query_hash', 'query_plan_hash') ORDER BY objName We can see that all of these are stored as binary(8)\nQuery Plan You can also find the Hashes in the Actual or Estimated Query Plan. They are also stored as binary(8), and you can find them either in the plan properties (of the main node).\nOr in the underlying XML.\nExtended Events For Extended Events, some events reference the Hashes directly.\n/* XE Event list */ SELECT dxp.name AS packageName , dxo.name AS eventName , dxoc.name AS columnName , dxoc.column_id , dxoc.type_name FROM sys.dm_xe_objects dxo JOIN sys.dm_xe_packages dxp ON dxo.package_guid = dxp.guid JOIN sys.dm_xe_object_columns AS dxoc ON dxoc.object_name = dxo.name AND dxoc.object_package_guid = dxo.package_guid AND dxoc.column_type \u003c\u003e N'readonly' WHERE dxo.object_type = 'event' AND dxoc.name LIKE 'query_%hash%' When an Extended Events event doesn't collect the Hashes by default, you can add them (where applicable) with the Global Fields/Actions.\n/* XE Global Action list */ SELECT CONCAT(dxp.name, '.', dxo.name) AS completeName , dxo.name AS ActionName , dxo.type_name , dxo.description FROM sys.dm_xe_objects dxo JOIN sys.dm_xe_packages dxp ON dxo.package_guid = dxp.guid WHERE dxo.object_type = 'action' and dxo.name LIKE 'query_%hash%' In the Actions and Events, we can see that the most common data type is int64 for the _signed version of the Hashes and uint64 for the unsigned. The int64 translates to the bigint SQL Server data type.\nSo which one should you use?\nTip Spoiler alert: always use the _signed version. The mapping When the data types don't match, you must convert at least one side.\nIf you match a Hash constant against a table, converting the constant is more efficient than converting the column.\nIt depends on the use case, so let's cover a few of them.\nCreate a test environment For our demo, we'll create three things\nA database with a Query Store enabled A procedure with several statements (so we can filter) An Extended Events monitor Here's a script that creates all three.\nCREATE DATABASE QueryHash ALTER DATABASE QueryHash SET QUERY_STORE = ON (QUERY_CAPTURE_MODE = ALL) GO USE QueryHash GO CREATE OR ALTER PROCEDURE dbo.FindMyHash AS BEGIN /* statement 1 */ SELECT * FROM sys.objects AS o WHERE o.create_date \u003e DATEADD(DAY, -1, GETDATE()) /* statement 2 */ ; -- previous stmt terminator WITH L0 AS(SELECT 1 AS c UNION ALL SELECT 1), L1 AS(SELECT 1 AS c FROM L0 CROSS JOIN L0 AS B), L2 AS(SELECT 1 AS c FROM L1 CROSS JOIN L1 AS B), Nums AS(SELECT ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) AS n FROM L2), tally AS (SELECT TOP (10) n FROM Nums ORDER BY n) SELECT n FROM tally /* statement 3 */ SELECT d.name , d.database_id AS dbId , d.log_reuse_wait_desc , d.compatibility_level AS compat , d.user_access_desc AS access , d.state_desc , d.snapshot_isolation_state_desc AS snpsht , d.is_read_committed_snapshot_on AS rcsi , d.recovery_model_desc , d.page_verify_option_desc AS pageVerify , d.is_fulltext_enabled AS fulltxt , d.is_trustworthy_on AS trstworthy , d.is_db_chaining_on AS dbChaining , d.is_query_store_on AS QS , d.is_broker_enabled AS brkr , d.is_cdc_enabled AS cdc , d.is_encrypted AS encr , d.delayed_durability_desc AS delayedDur , d.is_result_set_caching_on , d.is_memory_optimized_enabled FROM sys.databases AS d END GO /* Create a XE monitoring session and start it */ CREATE EVENT SESSION [FindMyHashXE] ON SERVER ADD EVENT sqlserver.sp_statement_starting ( SET collect_object_name=1 , collect_statement=1 ACTION ( sqlserver.query_hash /* to showcase the difference */ , sqlserver.query_hash_signed , sqlserver.query_plan_hash /* to showcase the difference */ , sqlserver.query_plan_hash_signed ) WHERE [object_name] = N'FindMyHash' ) GO ALTER EVENT SESSION FindMyHashXE ON SERVER STATE = START binary(8) to bigint The XE session FindMyHashXE is started, but we don't persist the data. Therefore we have to open the Watch Live Data XE Watch Live Data Once you have an XE session running, you can stream its events live in SSMS without needing a file target. Right-click the session in Object Explorer and choose Watch Live Data.\nCaveats:\nNo memory, no history. There is no target backing the session, so you can only see events that fire while you are actively watching. Anything that happened before you opened the live view is gone. The session keeps running. Closing the Watch Live Data tab does not stop the session. It is still capturing and discarding events in the background - like a tree falling in a forest with no one around to hear it. Stop or drop the session explicitly when you are done. viewer to collect the events.\nNext, let's run the Procedure couple of times.\nEXEC dbo.FindMyHash GO 10 /* GO 10 = repeats the batch 10 times */ Let's say I'm interested in the second statement.\nI could grab the Hashes from the Actual Execution Plan (Estimated doesn't really work for the Stored Procedure call).\nHere's the XML excerpt. The QueryHash and QueryPlanHash attributes are in the middle.\n\u003cStmtSimple StatementCompId=\"4\" StatementEstRows=\"10\" StatementId=\"2\" StatementOptmLevel=\"FULL\" StatementOptmEarlyAbortReason=\"GoodEnoughPlanFound\" CardinalityEstimationModelVersion=\"150\" StatementSubTreeCost=\"9.7468E-05\" StatementText=\"WITH\u0026#xD;\u0026#xA;\tL0 AS(SELECT 1 AS c \u003cshortened\u003e…\" StatementType=\"SELECT\" QueryHash=\"0x332D6151BE597D59\" QueryPlanHash=\"0x00DDF16427C45420\" RetrievedFromCache=\"true\" StatementSqlHandle=\"0x090042CC626E212F5B6C7B803CDB7A69839A0000000000000000000000000000000000000000000000000000\" DatabaseContextSettingsId=\"2\" ParentObjectId=\"581577110\" StatementParameterizationType=\"0\" SecurityPolicyApplied=\"false\" \u003e Copy out the Hashes manually and CAST them to bigint.\nSELECT CAST(0x332D6151BE597D59 AS bigint) AS query_hash_bigint, CAST(0x00DDF16427C45420 AS bigint) AS query_plan_hash_bigint /* 3687710673600085337 and 62471382319256608 respectively */ Or even better, we can grab all the information straight from the Query Store.\nSELECT qsq.query_id , qsp.plan_id , qsq.query_hash , CAST(qsq.query_hash AS bigint) AS query_hash_bigint , qsp.query_plan_hash , CAST(qsp.query_plan_hash AS bigint) AS query_plan_hash_bigint , qsqt.query_sql_text FROM sys.query_store_query AS qsq JOIN sys.query_store_plan AS qsp ON qsp.query_id = qsq.query_id JOIN sys.query_store_query_text AS qsqt ON qsqt.query_text_id = qsq.query_text_id WHERE qsq.object_id = OBJECT_ID(N'dbo.FindMyHash') Now we can go back to the Extended Events view and filter on the query_hash_signed column using the bigint value 3687710673600085337.\nWe opted to collect all executions in this demo and then filter them afterwards. But if you want to capture only a specific hash, you can add it as a filter in the XE session definition.\nLet's try the other scenario.\nBigint to binary(8) The tables have turned. Now you've noticed some problems with a query in the XE monitor, and you want to correlate with the Query Store to get more information. First, we'll clear the XE filter, and this time we'll grab the query_plan_hash_signed (7889493896768043449) for the third statement.\nWe'll plug this value into the query from above. But since it's not worth converting the whole column to match against a constant, we'll slightly update the statement.\nSELECT qsq.query_id , qsp.plan_id , qsq.query_hash , CAST(qsq.query_hash AS bigint) AS query_hash_bigint , qsp.query_plan_hash , CAST(qsp.query_plan_hash AS bigint) AS query_plan_hash_bigint , qsqt.query_sql_text FROM sys.query_store_query AS qsq JOIN sys.query_store_plan AS qsp ON qsp.query_id = qsq.query_id JOIN sys.query_store_query_text AS qsqt ON qsqt.query_text_id = qsq.query_text_id WHERE qsp.query_plan_hash = CAST(7889493896768043449 AS binary(8)) But wait! We didn't get back any result.\nAnd the reason is the Data Type Inference, as you can see in this demo.\nDECLARE @bigintHash bigint = 7889493896768043449 SELECT CAST(@bigintHash AS binary(8)) AS bigintToBin, /* 0x6D7D1CE61678B9B9 */ CAST(7889493896768043449 AS binary(8)) AS constToBin /* 0x13000001B9B97816 */ We pass the same value in both cases but get two different binaries. One type is explicitly defined, while the other is inferred.\nSo what is this mysterious data type? To find out, I'll use my favourite data type - sql_variant.\nDECLARE @test AS sql_variant = 7889493896768043449 SELECT SQL_VARIANT_PROPERTY(@test, 'BaseType') AS BaseType , SQL_VARIANT_PROPERTY(@test, 'Precision') AS Precision , SQL_VARIANT_PROPERTY(@test, 'Scale') AS Scale , SQL_VARIANT_PROPERTY(@test, 'TotalBytes') AS TotalBytes , SQL_VARIANT_PROPERTY(@test, 'MaxLength') AS MaxLength , SQL_VARIANT_PROPERTY(@test, 'Collation') AS Collation , CAST(@test AS binary(8)) AS variantToBin We can see that the underlying data type is numeric(19,0). Casting it to binary(8) returns the same binary value as in the example with the constant.\nSo how to fix our Query Store mapping? There are two options.\nUse a bigint variable and store the value there.\nDECLARE @queryPlanHash bigint = 7889493896768043449 SELECT … WHERE qsp.query_plan_hash = CAST(@queryPlanHash AS binary(8)) Or if you want to do it in a single statement - explicitly cast the constant as a bigint.\nSELECT … WHERE qsp.query_plan_hash = CAST(CAST(7889493896768043449 AS bigint) AS binary(8)) Both of these will get you the result.\nThe unsigned int I don't know if there is a use case for this, but it deserves mention for the sake of completeness.\nYou can find the detailed information in the article Correlating XE query_hash and query_plan_hash to sys.dm_exec_query_stats….\nI'll do a simplified version with just the calculation. I will pick the first statement because I need an example where query_hash \u003c\u003e query_hash_signed. As a reminder - these are our values:\nfield value query hash binary(8) 0xBA6ED813C4878164 query hash (unsigned) 13433912317905961316 query hash signed -5012831755803590300 I don't recommend this because we have to do the conversion on both sides of the mapping. We have to:\nRemove the most significant bit (MSB) from the binary value. Convert the unsigned number to a bigint. To remove the MSB we'll use the Bitwise AND (\u0026) with the max bigint value which is (2^63)-1 (9,223,372,036,854,775,807) and can also be represented in binary as 0x7FFFFFFFFFFFFFFF.\nBecause of this\nIn a bitwise operation, only one expression can be of either binary or varbinary data type.\nwe'll convert one side of the equation to bigint, and due to Data type precedence, that will also be the resulting data type.\nAs for the unsigned query hash, we'll subtract 2^63 (9,223,372,036,854,775,808), the magnitude of the smallest bigint value.\nTo put it all together.\nSELECT 0xBA6ED813C4878164 AS bin , CAST(0xBA6ED813C4878164 AS bigint) \u0026 0x7FFFFFFFFFFFFFFF AS binWithoutMSBtoBigint , 13433912317905961316 AS unsignedBigint , 13433912317905961316 - 9223372036854775808 AS signedBigint , CASE WHEN CAST(0xBA6ED813C4878164 AS bigint) \u0026 0x7FFFFFFFFFFFFFFF = 13433912317905961316 - 9223372036854775808 THEN 1 ELSE 0 END AS isSame Plugging this equation back into our Query Store statement would look like this.\nSELECT … WHERE CAST(qsq.query_hash AS bigint) \u0026 0x7FFFFFFFFFFFFFFF = (13433912317905961316 - 9223372036854775808) Which works, but it's just terrible for the performance and readability. So please stick to the signed version of the Hashes!\n","permalink":"/posts/query-hash-and-query-plan-hash-mapping/","tags":["Query Store","Extended Events"],"title":"Query Hash and Query Plan Hash Mapping"},{"categories":["Deep Dive"],"contents":"Foreword I always wondered what the KEEP PLAN hint does. The documentation isn't very specific (emphasis mine):\nForces the Query Optimizer to relax the estimated recompile threshold for a query.\nI'd like to prove what it really does.\nThe answer Like Dwarves of Moria, I delved too greedily and too deep. I found the answer in the Plan Caching and Recompilation in SQL Server 2012 whitepaper before trying it out on my own. Relevant excerpt:\nKEEP PLAN\nThe KEEP PLAN query hint changes the recompilation thresholds for temporary tables, and makes them identical to those for permanent tables. Therefore, if changes to temporary tables are causing many recompilations, this query hint can be used.\nBut since it doesn't show the proof, I decided to test it out anyway.\nRecompilations and AUTO_UPDATE_STATISTICS First, we need to cover these concepts for our test to make sense. According to Extended Events, there are 20 reasons for recompilations.\nSELECT dxmv.map_key, dxmv.map_value FROM sys.dm_xe_map_values AS dxmv WHERE dxmv.name = 'statement_recompile_cause' map_key map_value 1 Schema changed 2 Statistics changed 3 Deferred compile 4 Set option change 5 Temp table changed 6 Remote rowset changed 7 For browse permissions changed 8 Query notification environment changed 9 PartitionView changed 10 Cursor options changed 11 Option (recompile) requested 12 Parameterized plan flushed 13 Test plan linearization 14 Plan affecting database version changed 15 Query Store plan forcing policy changed 16 Query Store plan forcing failed 17 Query Store missing the plan 18 Interleaved execution required recompilation 19 Not a recompile 20 Multi-plan statement required compilation of alternative query plan I'm primarily interested in #2 - Statistics changed.\nWhat happens is described nicely in the AUTO_UPDATE_STATISTICS documentation. Again, emphasis mine:\nSpecifies that Query Optimizer updates statistics when they're used by a query and when they might be out-of-date. Statistics become out-of-date after insert, update, delete, or merge operations change the data distribution in the table or indexed view. Query Optimizer determines when statistics might be out-of-date by counting the number of data modifications since the last statistics update and comparing the number of modifications to a threshold. The threshold is based on the number of rows in the table or indexed view.\nQuery Optimizer checks for out-of-date statistics before it compiles a query and runs a cached query plan. Query Optimizer uses the columns, tables, and indexed views in the query predicate to determine which statistics might be out-of-date. Query Optimizer determines this information before it compiles a query. Before running a cached query plan, the Database Engine verifies that the query plan references up-to-date statistics.\nThe recompilation thresholds are described here. There has been a change in Compatibility Level (CL) 130, so I put it in the same table side by side for comparison.\nTable type Table cardinality (n) Pre CL130 # modifications to trigger recompilation Post CL130 # modifications to trigger recompilation Temporary or Permanent n \u003e 500 500 + (0.20 * n) MIN(500 + (0.20 *n), SQRT(1,000 * n)) Permanent n \u003c= 500 500 500 Temporary 6 \u003c= n \u003c= 500 500 500 Temporary n \u003c 6 6 6 Armed with this knowledge, let's test.\nTest scenario I will do the tests only on the CL150 environment as it includes both the old and new thresholds.\nI'll test the Permanent and Temporary thresholds separately because there are subtle differences.\nI'll create an environment with 3 different table sizes and 3 separate Stored Procedures that read from them. Each table size will test one of the thresholds (500, 20% and SQRT(1,000 * n)) Then I'll prepare monitoring to track the modification to stats, cached plans and recompilations. The first part of the test includes running the Procedures without the KEEP PLAN hint to verify the thresholds. Do a fresh update of the Statistics Run the Procedure to cache the plan Modify the Statistics just below the threshold Rerun the Procedure to see a cache hit Do one more modification Rerun the Procedure to see auto_stats and recompilation events Then I'll rerun the tests with the KEEP PLAN hint and see if the thresholds are affected. Finally, I'll repeat the tests on a statement using Temporary tables. Preparing the environment and monitoring Warning When testing this on my local instance running on Windows, the plan cache was randomly cleared, which affects the testing. Therefore I ran all my tests in a Docker container which didn't have this problem. The AUTO_UPDATE_STATISTICS must be enabled on the test DB and tempdb.\nCREATE DATABASE KeepPlan GO USE KeepPlan ALTER DATABASE KeepPlan SET AUTO_UPDATE_STATISTICS ON /* Helper table to fill and update the tables under test */ CREATE TABLE dbo.Nums ( n int , CONSTRAINT PK_Nums PRIMARY KEY (n) ) CREATE TABLE dbo.PermaSmall ( n int , CONSTRAINT PK_PermaSmall PRIMARY KEY (n) ) GO CREATE TABLE dbo.PermaMedium ( n int , CONSTRAINT PK_PermaMedium PRIMARY KEY (n) ) GO CREATE TABLE dbo.PermaLarge ( n int , CONSTRAINT PK_PermaLarge PRIMARY KEY (n) ) ;WITH L0 AS(SELECT 1 AS c UNION ALL SELECT 1), L1 AS(SELECT 1 AS c FROM L0 CROSS JOIN L0 AS B), L2 AS(SELECT 1 AS c FROM L1 CROSS JOIN L1 AS B), L3 AS(SELECT 1 AS c FROM L2 CROSS JOIN L2 AS B), L4 AS(SELECT 1 AS c FROM L3 CROSS JOIN L3 AS B), L5 AS(SELECT 1 AS c FROM L4 CROSS JOIN L4 AS B), Nums AS(SELECT ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) AS n FROM L5) , tally AS (SELECT TOP (100000) n FROM Nums ORDER BY n) INSERT INTO dbo.Nums WITH(TABLOCKX) (n) SELECT n FROM tally INSERT INTO dbo.PermaLarge WITH(TABLOCKX) (n) SELECT TOP (100000) n FROM dbo.Nums ORDER BY n INSERT INTO dbo.PermaMedium WITH(TABLOCKX) (n) SELECT TOP (1000) n FROM dbo.Nums ORDER BY n INSERT INTO dbo.PermaSmall WITH(TABLOCKX) (n) SELECT TOP (500) n FROM dbo.Nums ORDER BY n /* Have a fresh update of the statistics */ UPDATE STATISTICS dbo.PermaSmall WITH FULLSCAN UPDATE STATISTICS dbo.PermaMedium WITH FULLSCAN UPDATE STATISTICS dbo.PermaLarge WITH FULLSCAN GO CREATE OR ALTER PROCEDURE dbo.ReadSmall AS BEGIN SELECT * FROM dbo.PermaSmall AS ps WHERE ps.n \u003e= (SELECT 1) --OPTION (KEEP PLAN) END GO CREATE OR ALTER PROCEDURE dbo.ReadMedium AS BEGIN SELECT * FROM dbo.PermaMedium AS ps WHERE ps.n \u003e= (SELECT 1) --OPTION (KEEP PLAN) END GO CREATE OR ALTER PROCEDURE dbo.ReadLarge AS BEGIN SELECT * FROM dbo.PermaLarge AS ps WHERE ps.n \u003e= (SELECT 1) --OPTION (KEEP PLAN) END The subquery in the WHERE clause is to avoid a Trivial Plan, which doesn't care about the Statistics update.\nI've also created a procedure per table because the Statistics must be \"interesting\". Otherwise, I wouldn't be able to trigger auto_stats along with the recompilation.\nMonitoring statistics We can periodically check the stats, modification counter and thresholds with this query.\nSELECT obj.name AS tableName , stat.name AS statName , CAST(sp.last_updated AS time) AS Last_update_time , sp.rows , sp.steps , sp.modification_counter AS modCounter , d.compatibility_level AS CL , ( 500. + (0.20 * sp.rows)) AS calcFlat , (SQRT(1000. * sp.rows)) AS calcSqrt , ca.lesser AS lesserOfTwo , CASE WHEN sp.rows \u003c= 500 THEN 500. ELSE CASE WHEN d.compatibility_level \u003c 130 THEN 500. + (0.20 * sp.rows) WHEN d.compatibility_level \u003e= 130 THEN ca.lesser END END AS threshold FROM sys.objects AS obj JOIN sys.stats AS stat ON stat.object_id = obj.object_id JOIN sys.stats_columns AS sc ON sc.object_id = stat.object_id AND sc.stats_id = stat.stats_id AND sc.stats_column_id = 1 JOIN sys.columns AS c ON c.object_id = obj.object_id AND c.column_id = sc.column_id CROSS APPLY sys.dm_db_stats_properties (stat.object_id, stat.stats_id) AS sp JOIN sys.databases AS d ON d.database_id = DB_ID() CROSS APPLY ( SELECT MIN(v.threshold) AS lesser FROM ( VALUES ( 500. + (0.20 * sp.rows)) , (sqrt(1000. * sp.rows)) ) v (threshold) ) ca WHERE obj.is_ms_shipped = 0 AND obj.name LIKE N'Perma%' ORDER BY sp.rows OPTION (RECOMPILE, KEEPFIXED PLAN) This is my output\nMonitoring recompilations I will use an Extended Events session to track the caching and recompilations. Here's the definition.\nCREATE EVENT SESSION [StatementCompilation] ON SERVER ADD EVENT sqlserver.auto_stats ( WHERE [package0].[not_equal_uint64]([status],'Loading stats without updating') AND ( [sqlserver].[like_i_sql_unicode_string]([statistics_list],N'%Perma%') OR [sqlserver].[like_i_sql_unicode_string]([statistics_list],N'%#Temp%') ) ), ADD EVENT sqlserver.sp_cache_hit ( SET collect_object_name=1 , collect_plan_handle=1 WHERE [sqlserver].[like_i_sql_unicode_string]([object_name],N'Read%') ), ADD EVENT sqlserver.sp_cache_insert ( SET collect_cached_text=1 , collect_database_name=1 , collect_plan_handle=1 ACTION ( sqlserver.sql_text , sqlserver.tsql_stack ) WHERE [sqlserver].[like_i_sql_unicode_string]([object_name],N'Read%') ), ADD EVENT sqlserver.sp_statement_starting ( SET collect_object_name=1 , collect_statement=1 WHERE [sqlserver].[like_i_sql_unicode_string]([object_name],N'Read%') ), ADD EVENT sqlserver.sql_statement_recompile ( SET collect_object_name=1 , collect_statement=1 ACTION ( sqlserver.query_hash_signed ) WHERE [sqlserver].[equal_i_sql_unicode_string]([sqlserver].[database_name],N'KeepPlan') AND ( [sqlserver].[like_i_sql_unicode_string]([statement],N'%(SELECT 0)%') OR [sqlserver].[like_i_sql_unicode_string]([object_name],N'Read%') ) ), ADD EVENT sqlserver.sql_statement_starting ( SET collect_statement=1 WHERE [sqlserver].[like_i_sql_unicode_string]([statement],N'%(SELECT 0)%') OR [sqlserver].[like_i_sql_unicode_string]([statement],N'%FROM dbo.Nums%') ) ALTER EVENT SESSION StatementCompilation ON SERVER STATE = START Monitoring the plan cache I will use this statement to query information about the plan cache and parse some interesting columns out of the XML:\nWITH XMLNAMESPACES ('http://schemas.microsoft.com/sqlserver/2004/07/showplan' AS ns) SELECT decp.refcounts , decp.usecounts , ca1.si.value('@Statistics', 'nvarchar(255)') AS StatisticsName , ca1.si.value('@ModificationCount', 'bigint') AS ModificationCount , ca1.si.value('@SamplingPercent', 'bigint') AS SamplingPercent , ca1.si.value('@LastUpdate', 'datetime2(3)') AS LastUpdate , decp.plan_handle , deqp.query_plan FROM sys.dm_exec_cached_plans AS decp CROSS APPLY sys.dm_exec_query_plan(decp.plan_handle) AS deqp CROSS APPLY deqp.query_plan.nodes('//ns:OptimizerStatsUsage') AS ca(cr) CROSS APPLY ca.cr.nodes('ns:StatisticsInfo') AS ca1(si) WHERE OBJECT_NAME(deqp.objectid,DB_ID('KeepPlan')) LIKE 'Read%' OPTION (RECOMPILE) Testing the thresholds without the KEEP PLAN Let's start the XE session StatementCompilation and Watch Live Data XE Watch Live Data Once you have an XE session running, you can stream its events live in SSMS without needing a file target. Right-click the session in Object Explorer and choose Watch Live Data.\nCaveats:\nNo memory, no history. There is no target backing the session, so you can only see events that fire while you are actively watching. Anything that happened before you opened the live view is gone. The session keeps running. Closing the Watch Live Data tab does not stop the session. It is still capturing and discarding events in the background - like a tree falling in a forest with no one around to hear it. Stop or drop the session explicitly when you are done. .\nFirst, we will clear the plan cache.\nDBCC FREEPROCCACHE Then execute the first procedure. Run it as a single statement within the batch and without the white space.\nEXEC dbo.ReadSmall In the Monitoring statistics, we saw that the threshold for the table PermaSmall was 500.\nSo we will do 499 modifications using this query:\n/* Modify Perma tables */ ;WITH nums AS (SELECT TOP (499) n FROM dbo.Nums ORDER BY n) UPDATE p SET p.n = p.n /* fake modification */ FROM dbo.PermaSmall AS p /* Change table PermaSmall, PermaMedium, PermaLarge */ JOIN nums AS n ON p.n = n.n OPTION (KEEPFIXED PLAN) GO I will use this query many times, the only difference being the referenced table and the number in the TOP clause.\nNote The KEEPFIXED PLAN hint is there to prevent unwanted triggering of auto stats update. Let's execute the Procedure again to see that it's still in the cache. Then run the modification again, but now swapping the TOP (499) for TOP (1) And finally, rerun the Procedure - this should trigger the auto stats update and recompilation. This is my output from the XE session.\nThe 1 , 2 and 3 are the executions of the procedure. After the first execution, I could see the update of 499 rows and that the plan was still in cache. After the second execution, I did yet another single modification. I can see that the cache had a hit again, but after sp_statement_starting, there is sql_statement_recompile with the cause Statistics changed. That triggers the Loading and updating of the PermaSmall table Statistics. After that, I see the sp_statement_starting again, but this time with state = Recompiled. Testing again with a slight variation I will repeat the test for the PermaMedium and PermaLarge, but first, let's revisit the whitepaper from earlier.\nThere is an interesting excerpt about data modification tracking and its effect on the Recompilation Threshold (RT).\nDuring query compilation, the query processor loads zero or more statistics defined on tables referenced in a query. These statistics are known as interesting statistics. For every table referenced in a query, the compiled query plan contains:\nA list of all of the statistics loaded during query compilation. For each such statistic, a snapshot value of a counter that counts the number of table modifications is stored. The counter is called colmodctr. A separate colmodctr exists for each table column (except computed non-persisted columns). The threshold crossing test - which is performed to decide whether to recompile a query plan - is defined by the formula: ABS( colmodctr(current) - colmodctr(snapshot)) ) \u003e= RT I was not able to verify this in my testing. Therefore, I will repeat the previous test with a slight variation. I will do, for example, 100 modifications to the statistics before I cache the plan. If I plug the PermaMedium threshold and the snapshot value into the equation above:\nABS(colmodctr(current) - 100) \u003e= 700 colmodctr(current) \u003e= 800 That means I should be able to trigger a recompile after 800 modifications.\nLet's clear the cache again with the DBCC FREEPROCCACHE. Run the modification code against PermaMedium with a value of 100. And then cache the plan for the ReadMedium stored procedure. We can check the plan cache for the snapshot value. The following steps should be the same as previously:\nDo 599 modifications (We already have 100 modifications, and the threshold should be 700) Rerun the Procedure - cache hit Do 1 extra modification Rerun the Procedure - auto stats update + recompilation. Here are my results:\nWe can see that I did the first 100 modifications before caching the plan.\nBut the stats were recompiled when hitting the original threshold of 700.\nThe last remaining test is the PermaLarge table. That's to showcase the new SQRT threshold (instead of the old 20%). I will only post the XE result:\nEverything went as expected. Once we crossed the threshold of 10,000, auto stats update and recompile were fired.\nAdd a KEEP PLAN hint I will change the definition of the three Stored Procedures by uncommenting the KEEP PLAN hint. Then I'll rerun the previous tests hoping that the recompilation thresholds have been \"relaxed\" and find out the new threshold values.\nHere are the results\nThe thresholds are exactly the same as they were without the hint.\nWe can conclude that the KEEP PLAN does not affect Permanent tables. But how about Temp tables? If you remember the quote from the whitepaper, it should change the Temporary tables threshold to be the same as permanent tables.\nTemp table thresholds This took me a bit longer to test because there are some subtle differences.\nFirst of all, I won't be running it in a Stored Procedure because of the added complexity of Temp table caching and deferred compilations.\nI create the Temp table in one batch and run the statement in another one.\nI do both in the same session because of the Temp table scope.\nCREATE TABLE #TempTable ( n int ) INSERT INTO #TempTable (n) SELECT TOP (5) n FROM dbo.Nums ORDER BY n The statement to be cached:\nGO SELECT * FROM #TempTable AS tt WHERE tt.n \u003e (SELECT 0) --OPTION (KEEP PLAN) GO I use this statement to check the #TempTable stats (must be run in the tempdb)\nUSE tempdb SELECT LEFT(obj.name, 15) AS objName , obj.object_id , sp.last_updated , sp.rows , sp.rows_sampled , sp.steps , sp.modification_counter , CASE WHEN sp.rows \u003c 6 THEN '6' WHEN sp.rows BETWEEN 6 AND 500 THEN '500' ELSE 'Same as perma table' END AS TempThreshold FROM sys.objects AS obj JOIN sys.stats AS stat ON stat.object_id = obj.object_id JOIN sys.stats_columns AS sc ON sc.object_id = stat.object_id AND sc.stats_id = stat.stats_id AND sc.stats_column_id = 1 JOIN sys.columns AS c ON c.object_id = obj.object_id AND c.column_id = sc.column_id CROSS APPLY sys.dm_db_stats_properties (stat.object_id, stat.stats_id) AS sp WHERE obj.is_ms_shipped = 0 AND obj.type = 'U ' ORDER BY sp.rows OPTION (RECOMPILE, KEEPFIXED PLAN) Right after loading 5 rows, the stats look like this:\nBut with a similar update statement, I couldn't reproduce the threshold recompilation. I have only 5 rows and need 6 modifications to trigger the recompile (according to Docs). So I thought maybe I'd do multiple passes over the same data.\nThis update did not work for me.\n/* Modify Temp tables */ ;WITH nums AS (SELECT TOP (5) n FROM dbo.Nums ORDER BY n) UPDATE t SET t.n = t.n /* fake modification */ FROM #TempTable AS t JOIN nums AS n ON t.n = n.n OPTION (KEEPFIXED PLAN) GO 30 The GO 30 will repeat the batch 30 times. The stats showed several modifications.\nAnd even in XE, I couldn't see auto stats update or recompile\nI've tried deleting and reinserting the rows, no result there either. I finally made it work by just inserting new rows.\nLet's restart the test - clear the cache and drop and recreate the Temp table. This time I will initialize it with 3 rows. It doesn't matter as long as it's below 6.\nDBCC FREEPROCCACHE DROP TABLE IF EXISTS #TempTable CREATE TABLE #TempTable ( n int ) INSERT INTO #TempTable (n) SELECT TOP (3) n FROM dbo.Nums ORDER BY n GO Run the same statement as previously to cache it. Then add 5 new rows (1 below the threshold) Run the statement to see there is no recompilation Add 1 more row Run the statement and see auto stats update and recompilation Here is the XE result\nNow that we can reproduce the recompilation threshold let's reset and rerun the test but this time, let's uncomment the KEEP PLAN hint.\nWe can see that after the 6 modifications, there was no recompile. I've inserted an additional 493 rows, bringing it to 499 total modifications. There was still no recompile.\nAdding another row carried it over the new threshold of 500 and finally triggered the auto stats.\nFinally, the KEEP PLAN hint actually changed something. Let's test for the larger Temp tables.\nConclusion But wait! Do you remember the threshold table from the beginning?\nOnce the Temp table has more than 5 rows, it has the same threshold as the Permanent table. That means that the KEEP PLAN hint has no effect again!\nKEEP PLAN - I hereby dub thee the worst query hint I know.\nBe right back; I have to create a Pull Request and fix the hint's description in the SQL Docs.\nEdit (2022/04/19): Here's the Pull Request.\nEdit (2022/08/05): The documentation fix has been merged!\nUpdate (2022/04/19): Paul solved another mystery for me.\nRemember when I couldn't force the Temp table recompilations with modifications only and had to insert new rows? The reason for this is an optimization called Reduced recompilations for workloads using temporary tables across multiple scopes.\nBecause I'm on Compatibility Level 150, I can turn it off with these trace flags.\nDBCC TRACEON (11036, -1) -- reduce recompilations DBCC TRACEON (11048, -1) -- reduce recompilations 150 For brevity, I'll only post the test results. The test case is the same as the initial Temp table test.\nHere's the Temp table stats modification counter after 5 updates. And here's the full Extended Events output. We can see that it recompiles after 6 modifications while keeping the cardinality of 5.\nThen I clear the proc cache, recreate the Temp table and repeat the test with the KEEP PLAN hint.\nThe Temp table stats modification counter after 499 updates. The full Extended Events output. Since I have only 5 rows in the Temp table, I've repeated a modification to one row 499 times. After that, I rerun the query to see the recompile hadn't been triggered. One more modification and rerunning the query triggers the auto_stats The weird thing is that even after that, the sql_statement_starting state is still Normal and not Recompiled.\nWarning Don't forget to turn off the trace flags. DBCC TRACEOFF (11036, -1) DBCC TRACEOFF (11048, -1) Acknowledgements Paul White (SQL Kiwi) , who gave me useful tips when I was stuck, and who wrote these fantastic blog posts:\nTemporary Table Caching Explained Temporary Table Caching in Stored Procedures And these articles:\nUnderstanding When Statistics Will Automatically Update Does statistics update cause a recompile? Auto update statistics threshold of temp table in stored procedure ","permalink":"/posts/keep-plan-demystified/","tags":["Extended Events","Performance"],"title":"KEEP PLAN Demystified"},{"categories":["Investigation"],"contents":"Foreword I've helped answer another question that appeared on the SQL Server Slack:\nAre timestamps in XE event files you view in SSMS local or server time?\nTo test this, I need a server in a different timezone than the client (SSMS). The quickest and easiest tool for that is containers, more specifically Docker. I've written before about why I reach for containers.\nDon't have Docker installed? Grab it from this guide. You can still follow along with the examples even without it.\nPreparing the environment To create a new SQL Instance, we will just run the following command.\nIf you don't have the latest SQL Server 2019 image, it will download it for you (roughly 1.5 GB).\ndocker run ` -e 'ACCEPT_EULA=Y' ` -e 'SA_PASSWORD=Password5' ` -e 'MSSQL_PID=Developer' ` -e TZ='America/Los_Angeles' ` -p 14338:1433 ` -d ` --name xeinstance ` mcr.microsoft.com/mssql/server:2019-latest Command explanation docker run The command to create and run the container -e Environment variables. The EULA and SA Password are required; TZ sets the timezone needed for our test -p port mapping in format host:container. I'm using a different port on the host in case you already have a local instance. If you are based in the LA timezone, you can pick a different one, for example, Asia/Singapore. -d Detached --name Name of the container for easy reference mcr.microsoft.com/mssql/server:2019-latest Image name:tag. In our case, the latest CU of SQL Server 2019 If you need more information, you can run docker run --help or visit the Documentation. If the container is created successfully, its ID will appear. You can also check with the docker ps -a command.\nAdditional Docker references No need for this now, but if you want to dig deeper into Docker, here are some useful links:\nMicrosoft SQL Server Docker Image Documentation of the mssql Environment variables Great Docker quickstart guide by Andrew Pruski (DBA From The Cold) Test in the SSMS Connect to localhost,14338 (or .,14338 if you don't like typing). Select SQL Server Authentication Login = sa Password = Password5 Open a new query window and run this to confirm the server time.\nSELECT SYSDATETIMEOFFSET() , GETUTCDATE() The timezone will match our config from earlier.\nWe don't need to create a new XE session as the system_health will be running by default. If you're new to capturing errors this way, see Investigating errors with Extended Events.\nTo open it, navigate to the Docker instance in Object Explorer.\nGo to Management\\Extended Events\\Sessions\\System_health and double-click the package0.event_file.\nWe can add an additional column to the default view as per the image instructions. I am in the GMT+1 timezone, so we can see that the event file view shows time local to the SSMS client.\nDigging deeper Let's look at the underlying data with the fn_xe_file_target_read_file DMF.\nSELECT TOP (10) ca.eventDataXml.value('event[1]/@timestamp', 'datetime2(3)') AS tmpstmp , ca.eventDataXml FROM sys.fn_xe_file_target_read_file('system_health*.xel', NULL, NULL, NULL) AS ef CROSS APPLY ( VALUES (CAST(ef.event_data AS xml)) ) ca(eventDataXml) ORDER BY ef.timestamp_utc DESC This gives us a sample of the events. There is only one underlying timestamp column, and it is stored in UTC. SSMS converts it to your local time on the fly.\nSo, back to the original question: the Extended Events file viewer shows time local to the SSMS client, not the server. The data on disk is always UTC.\n","permalink":"/posts/extended-events-timezones/","tags":["Extended Events","Docker"],"title":"Extended Events Timezones"},{"categories":["Non-technical"],"contents":"Background It's a dangerous business, Frodo, going out your door. — Frodo (quoting Bilbo Baggins) (The Lord of the Rings) I've never been to an in-person conference. I haven't been to many virtual ones either - the major ones I've been to are SQLBits 2020 and PASS Data Community Summit 2021.\nSo I thought, when having a new experience, why not make it double trouble and try speaking for the first time as well. So I asked my mentor, Tracy Boggiano , to help me with a session, and we submitted it together. Sadly, Tracy had to cancel at the last minute due to unforeseen circumstances, so it was just me delivering it.\nTraveling We have till dawn. Then we must ride. — Aragorn (The Lord of the Rings) I'm not too fond of travelling. London isn't that far away, but I still had to wake up before 5 am, take a bus, train, bus again, plane, Tube, DLR and some walking and waiting to get to the hotel.\nI liked London public transport because I didn't have to change to local currency or buy a paper ticket. Just tap in and tap out with the debit card, and it figured out the price for you. Thanks to the community on SQL Server Slack for the travel tips. I'll skip the info about the hotel or the venue.\nOpening of the conference A wizard is never late, Frodo Baggins. Nor is he early; he arrives precisely when he means to. — Gandalf (The Lord of the Rings) The opening day had a few minor issues. First, the projected start was at 9 am, but the gate opened at 10:15. But the conference newcomers were encouraged to attend a Bit Buddy welcome session planned at 8, meaning they had to wait for the longest.\nThere were additional problems with the conference wifi, which meant people were switching to the free ExCeL wifi or creating personal hotspots, which (you've guessed it) made the connection problems even worse.\nSo with the late start, my first day training session had to make up for the lost time. But alas, it was very internet-dependent, and the problems with the wifi caused issues to the session flow.\nWhile the first day had a rocky start, organizing and building an event like this must be incredibly hard. However, the rest of the conference more than made up for it.\nPlanning the schedule All we have to decide is what to do with the time that is given us. — Gandalf (The Lord of the Rings) My strategy was to go through the app and pick the sessions - first by title, then by abstract. The session topics seemed well balanced by the organizers. I skipped all the Power BI/data analysis and most Azure sessions because they are not that relevant to me.\nI enjoyed the tags used for filtering the sessions. However, I can imagine it could be improved further by combined filtering. For example, only look for Intermediate sessions with the How tag and not Azure specific.\nIt was essential to plan the schedule because the venue was large, and you had to find your way to the next room.\nJust so you have an idea of the scale, here is the layout that I grabbed from CrowdCompass. But the sessions were only part of the conference value. I also wanted to meet the people - something that's not as good in the virtual space (even with the SpatialChat).\nNetworking Not all those who wander are lost. — Bilbo Baggins (The Lord of the Rings) The best times I had were between the sessions - in the community zone or around the booths.\nEveryone was super friendly and polite, never turning me down. However, it was the first larger in-person conference for many regulars, and they probably wanted to catch up with their friends after the two-year-long pause.\nThis situation made it harder to approach the SQL Server \"celebrities\" without feeling like I was bothering them.\nIt was also challenging to find people with the same specialization. It felt like I only talked with the Power BI folks.\nMaybe some form of colour-coding (lanyards, delegate cards, t-shirts, etc.) could help find like-minded people.\nMy session Even the smallest person can change the course of the future. — Galadriel (The Lord of the Rings) Lead up I tried to lure additional people to my session using social media's power with some game-themed self-promotion. Here are all the Tweets in chronological order; I had a great time coming up with those:\nView on Twitter View on Twitter View on Twitter View on Twitter View on Twitter View on Twitter Presenting As you can probably tell from the photo, I was extremely nervous. I slept like a baby the night before - I cried all night and soiled myself 🥁.\nThe nervousness came mainly from not having enough time, so I had to follow a very tight schedule. Any blunder or stutter (English is my second language) could set me back, and I wouldn't be able to catch up. Two tips that I got from previous sessions helped in particular:\nPractice the first 5 minutes over and over again until it's automatic (kudos to Cathrine Wilhelmsen ) Make a checklist from Kevin Kline - I made one to set up my demo correctly. I think I had at least 12 attendees, but it was hard seeing them with the lights in my face. No idea how many people joined online.\nAll in all, I think it went well. I only had one major blunder for a demo-heavy session, but I think I recovered nicely. We'll see on the replay and in the feedback forms.\nThe first feedback I got was from Grant Fritchey (Scary DBA) (a fellow Extended Events enthusiast), who heard about it and volunteered to go there - for which I'm grateful.\nLessons learned I thought the 20-minute sessions were more beginner-friendly, but it seems one of the hardest. Just welcoming the attendees and introducing yourself can take up to 3 minutes. Add another 2-4 minutes for the wrap-up, questions, feedback, and so on. Not a lot of time.\nI overestimated how much content I could fit in the sessions when writing the abstract. I couldn't change it too much after the realization - I wouldn't keep the promises made in the text.\nNext time I would go for either a longer session or less content.\nThe entertainment Gentlemen, we do not stop till nightfall. — Aragorn (The Lord of the Rings) There was a board game night, pub quiz night and costume fancy dress party. I spent each of these three activities with a different group of people and had a blast.\nI mostly looked forward to the party because getting the costume ready took some time and effort.\nFrom the very beginning, when the theme was unveiled, I knew that the centrepiece of my costume would be a face mask (at the time, Covid rules were more strict in the UK). So Mortal Kombat had plenty of characters that could do that - I just picked my favourite.\nI'm not the one for loud music, so I was happy there was a quiet area, which I promptly used. I spent most of the night chatting with David Wiseman about the open-source monitoring tool DBA Dash - a time well spent!\nParting thoughts Deeds will not be less valiant because they are unpraised. — Aragorn (The Lord of the Rings) First, a huge thank you to the organizers, helpers and everyone else involved. You did an outstanding job, and I enjoyed my time there. I can 100% recommend the experience, and I would gladly come back next year if I have the chance.\nWell, here at last, dear friends, on the shores of the Sea comes the end of our time at SQLBits. Go in peace!\n","permalink":"/posts/sqlbits2022/","tags":["sqlbits","Personal"],"title":"The SQLBits 2022, or There and Back Again"},{"categories":["How to"],"contents":"Foreword Continuing the series: you can't cheat at a game you cannot play. This time I've focused on the scoring algorithm. I have a hunch scoring will be useful for finding the optimal strategy.\nWhen I was done, I thought it would be fun to add some input and scoring validation to simulate the original game.\nWarning Disclaimer: This solution builds on the scripts created in the first part of this series. Strategy From the start, I wanted a set-based solution: split the words table so each letter sits in its own row. That way I can JOIN or INTERSECT them and generally have more options.\nDROP TABLE IF EXISTS dbo.WordLetter CREATE TABLE dbo.WordLetter ( WordId smallint NOT NULL , Word char(5) NOT NULL /* so I don't have to join */ , Letter char(1) NOT NULL , Position tinyint NOT NULL , CONSTRAINT PK_WordLetter PRIMARY KEY CLUSTERED (Letter, Position, WordId) , INDEX IX_WordLetter_Position NONCLUSTERED (Position) ) /* Unpivot the table */ INSERT INTO dbo.WordLetter WITH (TABLOCKX) (WordId, Word, Letter, Position) SELECT aw.Id , aw.Word , SUBSTRING(aw.Word, cj.Position, 1) , cj.Position FROM dbo.AllWords AS aw CROSS JOIN (VALUES (1), (2), (3), (4), (5)) AS cj(Position) Duplicate letters One problem I ran into in my earlier solutions was duplicate letters. I tested and debugged it using this example: the solution is REBUS and my guess is… GUESS.\nOnly one S is consumed and that's the one in 5th position because it's a perfect match. No other S remains in the solution and that's why it's blacked out.\nSince I'm only interested in the best match, I can join the two words on their shared letters and calculate the pairing distance like so.\nSELECT g.Word , g.Letter , g.Position , s.Word , s.Letter , s.Position , ABS(CAST(g.Position AS smallint) - s.Position) AS distance FROM dbo.WordLetter AS g /* guess */ JOIN dbo.WordLetter AS s /* solution */ ON g.Letter = s.Letter WHERE g.Word = 'guess' AND s.Word = 'rebus' ORDER BY g.Position Everything with distance = 0 is a perfect match. Any distance greater than 0 is a partial match, and everything else (not in the result) is not a match. I can partition by letter, order by distance and grab only the first row.\nHere's the final scoring algorithm, wrapped in a function:\nCREATE OR ALTER FUNCTION dbo.ScoreWordle ( @guessId smallint , @solutionId smallint ) RETURNS TABLE AS RETURN WITH score AS ( SELECT g.WordId AS guessWordId , s.WordId AS SolutionWordId , g.Letter , g.Position AS guessPosition , s.Position AS solutionPosition , ca.dist , ROW_NUMBER() OVER (PARTITION BY s.WordId, s.Letter, s.Position ORDER BY ca.dist) AS rn FROM dbo.WordLetter AS g JOIN dbo.WordLetter AS s ON g.Letter = s.Letter CROSS APPLY (VALUES (ABS(CAST(g.Position AS smallint) - s.Position))) AS ca(dist) WHERE g.WordId = @guessId AND s.WordId = @solutionId ) SELECT CONCAT ( MAX(CASE WHEN s.guessPosition = 1 THEN ca.score ELSE 'B' END) , MAX(CASE WHEN s.guessPosition = 2 THEN ca.score ELSE 'B' END) , MAX(CASE WHEN s.guessPosition = 3 THEN ca.score ELSE 'B' END) , MAX(CASE WHEN s.guessPosition = 4 THEN ca.score ELSE 'B' END) , MAX(CASE WHEN s.guessPosition = 5 THEN ca.score ELSE 'B' END) ) AS score FROM dbo.WordLetter AS wl LEFT JOIN score AS s ON wl.WordId = s.guessWordId AND s.rn = 1 AND s.guessPosition = wl.Position CROSS APPLY ( VALUES ( CASE WHEN s.dist = 0 THEN 'G' /* Green = match letter and position */ WHEN s.dist \u003e 0 THEN 'O' /* Orange = correct letter, wrong position */ ELSE 'B' /* Black = does not contain the letter */ END ) ) ca (score) WHERE wl.WordId = @guessId GROUP BY wl.WordId The complete game Again, the whole code is also in my GitHub repo. I'm creating several helper objects:\nGameHistory - table to hold the attempts and scoring PlayWordle - a stored procedure that validates the input and provides the scoring for our guesses ResetGame - another stored procedure to reset a single or all games The PlayWordle procedure runs these checks:\nThe guessed word must be on the list of available words Either a date or a Wordle number must be provided to match the relevant solution When neither is provided - today's date is picked The correct solution hasn't been guessed yet Max number of attempts hasn't been reached yet DROP TABLE IF EXISTS dbo.GameHistory CREATE TABLE dbo.GameHistory ( SolutionId smallint NOT NULL , GuessAttempt tinyint NOT NULL , GuessId smallint NOT NULL , Score char(5) NOT NULL , CONSTRAINT PK_GameHistory PRIMARY KEY CLUSTERED (SolutionId, GuessAttempt) ) CREATE OR ALTER PROCEDURE dbo.PlayWordle ( @guess char(5) , @date date = NULL , @wordleNum smallint = NULL /* has precedence over @date */ ) AS BEGIN SET XACT_ABORT ON BEGIN TRY /* Input handling */ DECLARE @guessId smallint , @solutionId smallint , @lastAttemptNum tinyint , @score char(5) /* Guessed word is valid */ SET @guessId = (SELECT Id FROM dbo.AllWords WHERE Word = @guess) IF (@guessId IS NULL) RAISERROR('Guessed word \"%s\" is invalid', 11, 1, @guess) /* Solution Id is provided either directly or through a date */ SET @solutionId = @wordleNum IF @solutionId IS NULL BEGIN SET @date = ISNULL(@date, GETDATE()) IF (@date \u003c '2021-06-19 00:00:00.000' OR @date \u003e '2027-10-21 00:00:00.000') RAISERROR('Date must be between 2021-06-19 and 2027-10-21', 11, 1) SET @solutionId = (SELECT s.Id FROM dbo.Solution AS s WHERE s.SolveDate = @date) END /* check if game is already finished */ IF EXISTS ( SELECT 1/0 FROM dbo.GameHistory WHERE SolutionId = @solutionId AND GameHistory.Score = 'GGGGG' ) BEGIN SELECT 'Please reset using \"EXEC dbo.ResetGame @wordleNum = ' + CAST(@solutionId AS varchar(5)) + '\".' AS [You already won] , gh.GuessAttempt , aw.Word , gh.Score FROM dbo.GameHistory AS gh JOIN dbo.AllWords AS aw ON gh.GuessId = aw.Id WHERE gh.SolutionId = @solutionId ORDER BY gh.GuessAttempt RETURN END /* Max number of attempts has not been reached yet */ SET @lastAttemptNum = ( SELECT MAX(GuessAttempt) FROM dbo.GameHistory WHERE SolutionId = @solutionId ) IF (@lastAttemptNum \u003e= 6) BEGIN DECLARE @maxMsg nvarchar(200) = CONCAT ( 'Solution #', @solutionId , ' already has 6 attempts. Please reset using ' , '\"EXEC dbo.ResetGame @wordleNum = ', @solutionId, '\".' ) RAISERROR(@maxMsg, 11, 1) END /* Calculate current guess score */ SELECT @score = sw.score FROM dbo.ScoreWordle(@guessId, @solutionId) AS sw /* Persist the guess attempt */ INSERT INTO dbo.GameHistory (SolutionId, GuessAttempt, GuessId, Score) SELECT @solutionId, ISNULL(@lastAttemptNum, 0) + 1, @guessId, @score SELECT gh.GuessAttempt , aw.Word , gh.Score FROM dbo.GameHistory AS gh JOIN dbo.AllWords AS aw ON gh.GuessId = aw.Id WHERE gh.SolutionId = @solutionId ORDER BY gh.GuessAttempt /* Check for solution or end game condition */ IF @score = 'GGGGG' SELECT CASE @lastAttemptNum + 1 WHEN 1 THEN 'Genius' WHEN 2 THEN 'Magnificent' WHEN 3 THEN 'Impressive' WHEN 4 THEN 'Splendid' WHEN 5 THEN 'Great' WHEN 6 THEN 'Phew' ELSE 'You won' END AS Victory IF (@score \u003c\u003e 'GGGGG' AND @lastAttemptNum = 5) SELECT s.Word AS Solution FROM dbo.Solution AS s WHERE s.Id = @solutionId END TRY BEGIN CATCH ;throw END CATCH END CREATE OR ALTER PROCEDURE dbo.ResetGame ( @wordleNum smallint , @deleteAll bit = 0 ) AS BEGIN IF @deleteAll = 1 TRUNCATE TABLE dbo.GameHistory ELSE DELETE FROM dbo.GameHistory WHERE SolutionId = @wordleNum END And you can play like this:\nEXEC dbo.ResetGame @wordleNum = 196 EXEC dbo.PlayWordle @guess = 'guess' , @wordleNum = 196 Enjoy the game!\nLooking forward I expected a longer break before any follow-up, partly because of sqlbits prep and partly because of the problem's complexity.\nI was never quite sure how to calculate the optimal guesses, and I'm still open to suggestions.\n","permalink":"/posts/sql-wordle-series-playing/","tags":["Wordle"],"title":"SQL Wordle Series - Part Two: Playing"},{"categories":["How to"],"contents":" Corrections 2026-05-29: The original Wordle site (powerlanguage.co.uk) is gone. The New York Times bought Wordle in 2022, so the \"view source\" trick below no longer works. The word lists still live in my GitHub repo, so you can follow along from there. One caveat: the date-based lookup matched the original word order, and the NYT has since edited the list, so it won't predict today's answer. Foreword You've probably heard about Wordle. Even though I'm late to the party, I'd like to post my take on this extremely popular word guessing game.\nWhile I'm not interested in playing the game myself, I'd like to find an optimal strategy to finish it in as few moves as possible. With this goal in mind, let's get started.\nOptimal strategy As the title says, the first part of this would-be series is the optimal strategy: cheating. Back when Wordle lived on powerlanguage.co.uk, you could open the page source and find a link to its script file. That file baked two word lists right into the game.\nI'll call the first list Solution and the second list Available. These lists don't overlap, as we can see on this Euler diagram.\nNow, the interesting thing about the Solution list is that it's ordered chronologically. When you know the start date, you can add n days, find the n-th element from the list, and get the solution for any given day.\nCreating the database I've extracted both lists and saved them into .txt files. Now we can create tables to hold them and import the data with BULK INSERT. I've put my lists in the D:\\SQL-Wordle-Series\\Lists\\ path. Alter your path accordingly.\nYou can find the two lists and the up-to-date DB creation script in my GitHub repo. Here is just the creation script:\nUSE [master] ALTER DATABASE [Wordle] SET SINGLE_USER WITH ROLLBACK IMMEDIATE GO DROP DATABASE [Wordle] GO CREATE DATABASE Wordle GO USE Wordle GO DROP TABLE IF EXISTS #Stage CREATE TABLE #Stage ( Word char(5) NULL ) DROP TABLE IF EXISTS dbo.Solution DROP TABLE IF EXISTS dbo.Available DROP SEQUENCE IF EXISTS dbo.WordId GO CREATE SEQUENCE dbo.WordId AS smallint START WITH 0 INCREMENT BY 1 MINVALUE 0 NO CYCLE CACHE 1000 CREATE TABLE dbo.Solution ( Id smallint NOT NULL CONSTRAINT DF_Solution_Id DEFAULT NEXT VALUE FOR WordId , Word char(5) NOT NULL UNIQUE , SolveDate AS DATEADD(DAY, Id, DATEFROMPARTS(2021, 06, 19)) , INDEX CX_Solution_Id CLUSTERED (Id) , CONSTRAINT PK_Solution_Word PRIMARY KEY NONCLUSTERED (Word) ) CREATE TABLE dbo.Available ( Id smallint NOT NULL CONSTRAINT DF_Available_Id DEFAULT NEXT VALUE FOR WordId , Word char(5) NOT NULL UNIQUE , INDEX CX_Available_Id CLUSTERED (Id) , CONSTRAINT PK_Available_Word PRIMARY KEY NONCLUSTERED (Word) ) BULK INSERT #Stage FROM 'D:\\SQL-Wordle-Series\\Lists\\Solution.txt' WITH ( FIRSTROW = 1, FIELDTERMINATOR = '', --CSV field delimiter ROWTERMINATOR = '\\n', --Use to shift the control to next row TABLOCK ) INSERT INTO dbo.Solution WITH (TABLOCKX) (Word) SELECT s.Word FROM #Stage AS s OPTION (MAXDOP 1) TRUNCATE TABLE #Stage BULK INSERT #Stage FROM 'D:\\SQL-Wordle-Series\\Lists\\Available.txt' WITH ( FIRSTROW = 1, FIELDTERMINATOR = '', --CSV field delimiter ROWTERMINATOR = '\\n', --Use to shift the control to next row TABLOCK ) INSERT INTO dbo.Available WITH (TABLOCKX) (Word) SELECT s.Word FROM #Stage AS s OPTION (MAXDOP 1) GO CREATE OR ALTER VIEW dbo.AllWords AS SELECT s.Id , s.Word , s.SolveDate FROM dbo.Solution AS s UNION ALL /* These two sets don't intersect */ SELECT a.Id , a.Word , NULL FROM dbo.Available AS a GO Find the solution Now we can easily query the Solution table either by the Id or by the SolveDate to find the previous and future solutions.\nSELECT s.* FROM dbo.Solution AS s WHERE s.SolveDate = DATEFROMPARTS(2022, 02, 07) --OR s.Id = 233 If you don't want to spoil your fair game streak, you can test it on the Wordle Archive website.\nLooking forward In the next post, I'll come up with a way to play Wordle inside SQL Server.\nAfter that, I'd hoped to tackle the hardest topic: finding an optimal (non-cheating) algorithm, and maybe even Hard mode.\n","permalink":"/posts/sql-wordle-series-cheating/","tags":["Wordle"],"title":"SQL Wordle Series - Part One: Cheating"},{"categories":["Investigation"],"contents":"How can 15 be less than 13? I saw this puzzle on Twitter and couldn't resist solving it myself.\nI was running SQL Server 2019 (major version 15). The query looked something like this:\nSELECT SERVERPROPERTY('ProductMajorVersion') AS MajorVersion, CASE WHEN SERVERPROPERTY('ProductMajorVersion') \u003c 13 THEN 'True' ELSE 'False' END AS LogicalTest This returns True. If it's the first time you're seeing a problem like this, it can be confusing. How can 15 be less than 13?\nMy first train of thought was the same as the original poster's. Ok, so both values are converted to varchar according to data type precedence - similar to this:\nSELECT CASE WHEN '12' \u003c '2' THEN 'True' ELSE 'False' END AS LogicalTest Notice the quotes around the number. I'm not comparing numbers but numbers stored as a varchar.\nBecause string comparison works on a character basis, '2' is larger than '1' (as 'B' would be larger than 'A'). The second character doesn't matter.\nBut my theory had three major flaws.\nFirst flaw '15' isn't smaller than '13'. The first character '1' is the same in both strings and second character '5' is larger than '3'.\nSELECT CASE WHEN '15' \u003c '13' THEN 'True' ELSE 'False' END AS LogicalTest Second flaw I hadn't checked the Data type precedence.\nThe int is actually higher than the varchar, so the varchar would be converted to int (if possible).\nSELECT CASE WHEN '15' \u003c 13 THEN 'True' ELSE 'False' END AS LogicalTest Third flaw I hadn't checked the return type of the SERVERPROPERTY function. It's a sql_variant and not varchar, which makes sense. It can have several different data types returned for different properties.\nAnd since the sql_variant type is almost at the top of the precedence list (one of the reasons it's my favourite data type), the int value will also be converted to sql_variant.\nThis has been a perfect storm of bad assumptions. Now, let's correct those.\nsql_variant comparison Like Asimov's Laws of Robotics, sql_variant also has three rules for comparison.\nTaken from the documentation (emphasis mine):\nFirst rule When sql_variant values of different base data types are compared and the base data types are in different data type families, the value whose data type family is higher in the hierarchy chart is considered the greater of the two values. — (Microsoft Docs) Much like nepotism - values don't matter, only (data type) families.\nLet's confirm the base data types of our values.\nSELECT SQL_VARIANT_PROPERTY ( SERVERPROPERTY('ProductMajorVersion') , 'BaseType' ) AS BaseTypeServerProperty , SQL_VARIANT_PROPERTY ( 13 , 'BaseType' ) AS BaseTypeNumber This returns nvarchar and int respectively. Now we have to match them with their data type family. I have aggregated the table so the data type families are unique (original table is in the documentation link). The data types in rows are also ordered.\n# Data type family Base data type 1 sql_variant sql_variant 2 Date and time datetime2, datetimeoffset, datetime, smalldatetime, date, time 3 Approximate numeric float, real 4 Exact numeric decimal, money, smallmoney, bigint, int, smallint, tinyint, bit 5 Unicode nvarchar, nchar, varchar, char 6 Binary varbinary, binary 7 Uniqueidentifier Uniqueidentifier The int is higher in the table than the nvarchar and is therefore considered greater of the two. I can repeat the test with an int and a datetime.\nDECLARE @dtVar sql_variant = GETDATE() DECLARE @intVar sql_variant = 9999999 SELECT CASE WHEN @intVar \u003c @dtVar THEN 'True' ELSE 'False' END AS LogicalTest Which evaluates to True. That proves it and with this, our mystery is solved!\nBut just for fun, let's do the remaining two rules.\nSecond rule When sql_variant values of different base data types are compared and the base data types are in the same data type family, the value whose base data type is lower in the hierarchy chart is implicitly converted to the other data type and the comparison is then made. — (Microsoft Docs) This doesn't apply to the Unicode family as per the rule below. So let's test with two different Exact numeric base types.\nDECLARE @bigInt sql_variant = CAST(1 AS bigint) DECLARE @intVar sql_variant = CAST(2 AS int) SELECT CASE WHEN @bigInt \u003c @intVar THEN 'True' ELSE 'False' END AS LogicalTest The int value is considered greater (because it is) than the bigint value even though the latter is higher in the data type table.\nNow it's actually comparing values.\nThird rule When sql_variant values of the char, varchar, nchar, or nvarchar data types are compared, their collations are first compared based on the following criteria: LCID, LCID version, comparison flags, and sort ID. Each of these criteria are compared as integer values, and in the order listed. If all of these criteria are equal, then the actual string values are compared according to the collation. — (Microsoft Docs) I won't do all of these combinations, I'll just try the first one - LCID.\nSELECT COLLATIONPROPERTY('French_CI_AI', 'LCID') SELECT COLLATIONPROPERTY('Latin1_General_CI_AS', 'LCID') DECLARE @French sql_variant = N'abc' COLLATE French_CI_AI DECLARE @English sql_variant = N'abc' COLLATE Latin1_General_CI_AS SELECT CASE WHEN @English \u003c @French THEN 'True' ELSE 'False' END AS LogicalTest The LCID for French is 1036 which is greater than 1033 for English. That proves @French is greater than @English.\nFinal thought To prevent these shenanigans, embrace defensive programming.\nTip If you have an assumption - either enforce it or verify it. In this case, I would explicitly CAST the SERVERPROPERTY to an int data type like so.\nSELECT SERVERPROPERTY('ProductMajorVersion') AS MajorVersion, CASE WHEN CAST(SERVERPROPERTY('ProductMajorVersion') AS int) \u003c 13 THEN 'True' ELSE 'False' END AS LogicalTest ","permalink":"/posts/expecting-subvertations/","tags":["Debugging"],"title":"Expecting Subvertations"},{"categories":["Investigation"],"contents":"The problem A colleague needed to find who was changing a specific cell in a table with thousands of modifications per minute. It could be an automated process, application logic, an application user or even an ad-hoc statement - we didn't know. Monitoring everything and sifting through the noise wasn't an option.\nWe wanted to learn the who, the how and then ask why? If you'd like to know the whole journey, read on. Otherwise, you can skip to the Eureka moment.\nPossible solutions Off the top of my head, I thought of these options:\nSQL Audit Data mine the Query Store to find the possible access patterns Extended Events (XE) - but what should I filter? Trigger on the table + logging table 1. SQL Audit I admit I haven't used Audit before (I just knew of its existence), so it was an opportunity to try it out. You need to:\nSet up a Server Audit Specification to choose the target file Set up a Database Audit Specification Select Audit Action Type, Object and Principal, and we are good to go On the positive side, it captures who and how.\nOn the negative side, it captures everything. If I have tens of thousands of modifications per hour, I have to collect everything and look for the culprit.\nThis wouldn't do.\n2. Data mine the Query Store Query Store has aggregated data, so I knew I wouldn't be getting any specific query execution. I was here to map out all the access patterns and perhaps pass the signed query hash to Extended Events.\nI didn't have a sophisticated method, just a full-text search on sys.query_store_query_text.\nI was looking for the table name along with the MERGE or UPDATE keywords.\nBut there were just too many hits - it would maybe get me closer, but I still wouldn't know specifics.\n3. Extended Events At this point, there was still nothing to filter. I could grab all the query hashes I'd gathered from mining the Query Store, but the update could come from a different query next time. I could collect everything and filter, but SQL Audit would do a better job.\n4. Trigger Triggers are often misused and get a bad reputation, but this is a perfectly fine use case. All I need is an AFTER UPDATE Trigger and a logging table - easy peasy. While the Trigger fires on each UPDATE, I'll have logic inside to only log information on certain conditions.\nI can use the HOST_NAME(), ORIGINAL_LOGIN() and other functions to capture the who info.\nWhat I was missing was how? Was it an ad-hoc statement, Stored Procedure or something else? I wanted to capture the input buffer. I thought there might be a function for that but couldn't find it.\nAll I needed was to query the DMVs. But that requires a VIEW SERVER STATE permission. That invalidates the Trigger, so I need to use the Module Signing. That means creating a certificate, creating a Login, etc., which felt like a lot of work for a simple auditing task. Especially if you do that infrequently. Or when you are new to Module Signing.\nIt's a shame! I already had the who figured out. I was halfway there!\nEureka moment Why do I have to use just one tool to finish the job? I wanted to combine the Query Store hash information with the Extended Events. I can instead combine it with the Trigger!\nWithin the Trigger, I have logic to filter the exact changes I'm interested in.\nAnd with the Extended Events, I can monitor only the statement in my Trigger logic and grab all the other auditing info - especially the input buffer and the tsql stack.\nHere's a diagram:\nFinally, I had both the who and how!\nThe proof Let's create a table and some data.\nDROP TABLE IF EXISTS dbo.PopularTable CREATE TABLE dbo.PopularTable ( Id int PRIMARY KEY , InterestingColumn int NOT NULL /* Monitor this column */ , OtherColumn int NOT NULL ) INSERT INTO dbo.PopularTable (Id, InterestingColumn, OtherColumn) VALUES (1, 0, 0) , (2, 0, 0) /* Monitor this row */ , (3, 0, 0) Next, I'll create an AFTER UPDATE Trigger on this table. Inside I'll check just updates to the row with Id = 2 and InterestingColumn. I also only want to alert when the value actually changes.\nCREATE OR ALTER TRIGGER dbo.PopularTableUpdateInterestingColumn ON dbo.PopularTable AFTER UPDATE AS BEGIN IF EXISTS ( SELECT InterestingColumn FROM Inserted WHERE Id = 2 EXCEPT SELECT InterestingColumn FROM Deleted WHERE Id = 2 ) BEGIN RETURN END END GO /* Set the trigger order to 'Last' for a good measure */ exec sp_settriggerorder @triggername = 'dbo.PopularTableUpdateInterestingColumn' , @order='Last' , @stmttype = 'UPDATE' Apart from the check, the Trigger doesn't do anything. We will set up an Extended Events session that will watch the execution of this Trigger but only the RETURN statement inside the IF - which has a query_hash_signed = 0. The RETURN has no query text to hash, so it always hashes to zero. And since there's no other executable code in the trigger body, this filter is unambiguous.\nCREATE EVENT SESSION [MonitorPopularTableTrigger] ON SERVER ADD EVENT sqlserver.sp_statement_starting ( SET collect_object_name = 1 ACTION ( server_instance_name , server_principal_name , client_app_name , client_hostname , client_pid , session_id , sql_text , tsql_stack , query_hash_signed ) WHERE [object_name] = N'PopularTableUpdateInterestingColumn' AND query_hash_signed = 0 ) For your use case, you'll want to add a file event target as well as tweak the collected actions.\nIn this case, Watch Live Data will do just fine.\nWith all the pieces arranged, all that is left is some access code and test scenarios.\nI'll create a stored procedure to run the first few test cases and then use ad-hoc UPDATE and MERGE statements.\nCREATE OR ALTER PROCEDURE dbo.UpdatePopularTable ( @Id int , @InterestingColumn int , @OtherColumn int ) AS BEGIN UPDATE dbo.PopularTable SET InterestingColumn = @InterestingColumn , PopularTable.OtherColumn = @OtherColumn WHERE Id = @Id END Let's call each test in separate batches, so our monitoring sql_text is clean. Don't forget we are only interested in row 2 InterestingColumn change.\nTest case 1 Change via Stored Procedure Update InterestingColumn Update row 1 Nothing should show up in XE /* Test 1 */ EXEC dbo.UpdatePopularTable @Id = 1 , @InterestingColumn = 1 , @OtherColumn = 0 Test case 2 Change via Stored Procedure Update OtherColumn Update row 2 Nothing should show up in XE /* Test 2 */ EXEC dbo.UpdatePopularTable @Id = 2 , @InterestingColumn = 0 , @OtherColumn = 1 Test case 3 Change via Stored Procedure Update InterestingColumn Update row 2 There should be a captured event /* Test 3 */ EXEC dbo.UpdatePopularTable @Id = 2 , @InterestingColumn = 1 , @OtherColumn = 0 Test case 4 Change via ad-hoc UPDATE statement Update InterestingColumn Update multiple rows including the row 2 There should be a captured event /* Test 4 */ ;WITH rowsForUpdate AS ( SELECT * FROM dbo.PopularTable WHERE Id BETWEEN 2 AND 3 ) UPDATE rowsForUpdate SET InterestingColumn = 2 Test case 5 Change via ad-hoc MERGE statement Update InterestingColumn Update row 2 Insert additional row - to justify MERGE over UPDATE There should be a captured event /* Test 5 */ MERGE INTO dbo.PopularTable AS tgt USING ( VALUES (2, 3, 0) , (5, 0, 1) ) AS src (Id, ic, oc) ON tgt.Id = src.Id WHEN MATCHED THEN UPDATE SET tgt.InterestingColumn = src.ic WHEN NOT MATCHED THEN INSERT (Id, InterestingColumn, OtherColumn) VALUES (src.Id, src.ic, src.oc); Results As expected - the first two tests resulted in no event captured. The other three are there with their respective sql_text. The OUTPUT Clause Trap There's one risk I've run into: adding a trigger can break unrelated code.\nError 334 says: The target table '%.*ls' of the DML statement cannot have any enabled triggers if the statement contains an OUTPUT clause without INTO clause. Let's see this in action.\nUPDATE dbo.PopularTable SET PopularTable.InterestingColumn = 1 OUTPUT Deleted.* WHERE PopularTable.Id = 2 This gives us the error above. The fix is to OUTPUT into a table variable and then read from it.\nDECLARE @outputTable AS TABLE ( Id int NOT NULL , InterestingColumn int NOT NULL , OtherColumn int NOT NULL ) UPDATE dbo.PopularTable SET PopularTable.InterestingColumn = 1 OUTPUT Deleted.* INTO @outputTable WHERE PopularTable.Id = 2 SELECT * FROM @outputTable AS ot Personally, I think removing the \"naked\" OUTPUT is the way to go, because there are more valid cases for adding triggers.\nNote The trigger fires on every UPDATE to the table, even when the filter doesn't match. The overhead is minimal - the EXCEPT check on Inserted/Deleted is lightweight - but it's nonzero. Also remember that TRUNCATE TABLE bypasses triggers entirely, though it's easier to trace and requires elevated permissions. Get alerted on new events Now, this is well beyond the scope of this article, but you can use dbatools PowerShell module to automate alerting. You can tweak this snippet to add a smart target to our existing XE session, and you'll get an email notification on the next event.\nNote This keeps an open connection to the server which listens to the XE session. $params = @{ SmtpServer = \"YourSmtp\" To = \"Recipient@Address.com\" Sender = \"MonitorPopularTableTrigger@powershell.com\" Subject = \"PopularTable Trigger fired on {server_instance_name}\" Body = \" Timestamp: {collection_time} Client Application name: {client_app_name} Hostname: {client_hostname} Login {server_principal_name} Text: {sql_text} Stack: {tsql_stack} \" } $emailresponse = New-DbaXESmartEmail @params Start-DbaXESmartTarget -SqlInstance 'YourInstance' ` -Session 'MonitorPopularTableTrigger' ` -Responder $emailresponse # Clean up all smart targets with Get-DbaXESmartTarget | Remove-DbaXESmartTarget ","permalink":"/posts/how-to-audit-data-modifications-with-surgical-precision/","tags":["Extended Events","Debugging","Security"],"title":"How to audit data modifications with surgical precision"},{"categories":["Investigation"],"contents":" Corrections 2021-12-10: Added the Clarification section to clarify that IS locks on Indexed Views are expected behavior, not a bug. Foreword I sometimes document SQL mysteries I've helped other people debug. This one happened to me.\nThe problem I was analyzing a deadlock graph and there was a mystery lock of type IS (Intent Shared). That was weird by itself because the database has RCSI enabled, which is the Optimistic Concurrency model that shouldn't take shared locks. All the statements were contained in this database. Also, the locked table was seemingly unrelated to anything that has been going on in the deadlock report.\nThe statement which took this block was a simple UPDATE query. So I checked the usual suspects:\nIs it a table or a View? Does it have any Triggers? Is there a Scalar function (in a computed column or constraint)? Is there a Foreign key to the unrelated table? It was none of those things. Then I looked at what those two tables had in common and the only thing I came up with was an Indexed View which turned out to be the problem.\nRepro Let's create a brand new database.\nCREATE DATABASE TestLock Inside of this database, we'll create two tables and fill them with sample data.\nCREATE TABLE dbo.MainTable ( Id INT PRIMARY KEY , PrivateColumn CHAR(1) NOT NULL , ColumnInView CHAR(1) NOT NULL ) CREATE TABLE dbo.UnrelatedTable ( Id INT PRIMARY KEY , RandomColumn CHAR(1) NOT NULL ) INSERT INTO dbo.MainTable (Id, PrivateColumn, ColumnInView) VALUES (1, 'P', 'V') , (2, 'P', 'V') INSERT INTO dbo.UnrelatedTable (Id, RandomColumn) VALUES (1, 'R') , (2, 'R') And the last object will be an Indexed View that combines two tables, but only exposes the ColumnInView from the MainTable.\nSET NUMERIC_ROUNDABORT OFF SET ANSI_PADDING, ANSI_WARNINGS, CONCAT_NULL_YIELDS_NULL , ARITHABORT, QUOTED_IDENTIFIER, ANSI_NULLS ON GO CREATE OR ALTER VIEW dbo.IndexedView WITH SCHEMABINDING AS SELECT ut.Id , ut.RandomColumn , mt.ColumnInView , COUNT_BIG(*) AS cnt /* indexed view requirement */ FROM dbo.UnrelatedTable AS ut JOIN dbo.MainTable AS mt ON mt.Id = ut.Id GROUP BY ut.Id , ut.RandomColumn , mt.ColumnInView GO CREATE UNIQUE CLUSTERED INDEX CX_IndexedView ON dbo.IndexedView (Id) Then we open a new session - let's call it a monitoring session that we'll watch with XE. This session will be reused so take note of the session_id (SPID) and do not close.\nYou can find the session_id either by running SELECT @@SPID, in the session tab info or the status bar.\nWe confirm that the database doesn't have RCSI enabled.\nSELECT is_read_committed_snapshot_on FROM sys.databases WHERE name = N'TestLock' Then we create the Extended Events session to monitor the locks. This is the scripted out definition. Change the SPID to match the SPID of your monitoring session.\nCREATE EVENT SESSION [LockAcquired] ON SERVER ADD EVENT sqlserver.lock_acquired ( SET collect_resource_description = 1 WHERE ( /* change to your SPID */ [package0].[equal_uint64]([sqlserver].[session_id],(52)) AND [resource_type] = 'OBJECT' AND [mode] \u003c\u003e 'SCH_S' ) ) GO ALTER EVENT SESSION [LockAcquired] ON SERVER STATE = START I'm only interested in the Object level locks. I also filter out the Schema stability locks (SCH_S). No target is required. Watch Live Data XE Watch Live Data Once you have an XE session running, you can stream its events live in SSMS without needing a file target. Right-click the session in Object Explorer and choose Watch Live Data.\nCaveats:\nNo memory, no history. There is no target backing the session, so you can only see events that fire while you are actively watching. Anything that happened before you opened the live view is gone. The session keeps running. Closing the Watch Live Data tab does not stop the session. It is still capturing and discarding events in the background - like a tree falling in a forest with no one around to hear it. Stop or drop the session explicitly when you are done. will do.\nIt's also useful to take note of the object IDs of our tables and the Indexed View. These are mine:\nSELECT o.object_id, o.name, o.type FROM sys.objects AS o WHERE o.is_ms_shipped = 0 AND o.type IN ('U', 'V') Now, let's run a simple select from the MainTable in our monitoring session while we have the XE session opened in another window.\nSELECT * FROM dbo.MainTable My XE output shows this:\nThere are two objects with IS locks - the 581577110 matches the MainTable. You might be wondering what is the other lock according to SELECT OBJECT_NAME(245575913) - it's a plan_persist_context_settings.\nWe can clear the data from the XE Live Data (via the Extended Events \u003e Clear Data menu), enable RCSI and repeat our experiment.\nUSE [master] ALTER DATABASE TestLock SET SINGLE_USER WITH ROLLBACK IMMEDIATE ALTER DATABASE TestLock SET READ_COMMITTED_SNAPSHOT ON ALTER DATABASE TestLock SET MULTI_USER Check again the RCSI status with the query we used previously. Run the SELECT on MainTable again in our monitoring session - the plan_persist_context_settings remains, but the MainTable IS lock is gone.\nEnter the Indexed view First I'll update the PrivateColumn of the MainTable and again watch the locks in the XE session.\nUPDATE dbo.MainTable SET MainTable.PrivateColumn = 'A' I'll disregard the plan_persist_context_settings. Then we have an X lock on the IndexedView and IX on the MainTable.\nNow let's update the ColumnInView, which is also referenced in the IndexedView.\nUPDATE dbo.MainTable SET MainTable.ColumnInView = 'B' Apart from the same locks we got last time, there is also an IX lock on the IndexedView but, more interestingly, an IS lock on the UnrelatedTable.\nThis confirms my theory that the Indexed View is the culprit of the deadlock graph from earlier. Indexed Views might add performance for reading but they hurt the concurrency and not even RCSI can save it.\nClarification It has been brought to my attention by Paul White (SQL Kiwi) that this post might seem like I'm describing a bug or unintended behaviour.\nTo make amends, I'm sharing this great post by Erik Darling (Darling Data) that covers it better and more in-depth: Locks Taken During Indexed View Modifications The links in that post are worth following too.\nThe point I was trying to make is that usually, the other \"hidden\" culprits (Triggers, Scalar functions in a table definition, Foreign keys) are tied directly to the table. But the Indexed View is truly hidden on the sidelines.\nSo here's a quick query to find an Indexed View and which tables/columns it's referencing:\nSELECT v.object_id , SCHEMA_NAME(v.schema_id) AS schemaName , v.name AS viewName , i.name AS indexName , dsre.referenced_schema_name AS refSchema , dsre.referenced_entity_name AS refObject , dsre.referenced_minor_name AS refColumn FROM sys.views AS v JOIN sys.indexes AS i ON i.object_id = v.object_id AND i.index_id = 1 /* Clustered */ CROSS APPLY sys.dm_sql_referenced_entities ( CONCAT ( OBJECT_SCHEMA_NAME(v.object_id) , '.' , v.name ) , N'OBJECT' ) AS dsre --WHERE v.name LIKE N'%%' /* Filter a specific view */ ","permalink":"/posts/is-lock-in-rcsi-enabled-database/","tags":["Extended Events","Debugging","Indexed View","Locking"],"title":"IS Lock in RCSI Enabled Database"},{"categories":["Opinion"],"contents":"Foreword I like to help people with their SQL problems. I frequent the SQL Server community Slack channel, DBA Stack Exchange, Microsoft Q\u0026A, etc. and challenge myself to answer questions - especially the ones where I don't know the answer. It's a good learning exercise.\nBut everyone should at least try to solve their problems first before asking for help. You cannot learn advanced debugging skills when you lack the basic ones. Here are the ones I keep running into.\n1. Let me Google that for you I further divide this into two subcategories.\nMicrosoft Documentation Q: What units does the total_elapsed_time column use in the sys.dm_exec_sessions DMV?\nAnswer: Highlight the sys.dm_exec_sessions in SSMS, press F1 and it goes to the documentation page - find or scroll down to total_elapsed_time and you can see it's in milliseconds.\nReputable sources For example blogs, highly upvoted questions on Stack Exchange, etc.\nHow to clear procedure cache but only for a single database?\nAnswer:\nOpen a search engine of your choice Prefix your search with mssql, tsql, sql server or some other keyword (so you don't get some other DB engine results) Get rid of the noise words It could look like this:\nAnd the results:\nDBA StackExchange - How to clear all plans from a single database? recommends DBCC FLUSHPROCINDB (\u003cdb_id\u003e) which I couldn't find in the official documentation.\nAnother example from Pinal Dave wrote about Cleanup Plan Cache for a Single Database: ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE.\nBoth methods also appear in Glenn Berry 's SqlSkills post on eight ways to clear the plan cache. Glenn is a very reputable source, and in time, you'll begin to recognize people providing trustworthy content.\n2. Can't be bothered to try As an example, I'll use the old database myth that has been busted many times:\nTruncate table is not logged / cannot be rolled back\nIt's not true and it can be easily tested.\nAs you can see in the video, even with typos and hesitations it took me only about 2 minutes to test and debunk this.\nBut what about Heap vs Clustered Index? Simple vs Full recovery model? Just test it. I've just shown you how easy it was.\nAnother question I have recently seen.\nDoes renaming the index change the stats name?\nAnswer:\nCreate a table Create an index - take a note of the stats name Rename the index - take a note of the stats name Compare I'll leave this as an exercise for the reader. You may not even know how to create a table or check the stats name. This is the beauty of it. You learn it this way, and what you've learned will stay with you longer.\n3. Interrogation Playing 20 questions with the asker to get the full picture of the problem. The XY Problem. Withholding crucial information. E.g. That table is not a table but a view, and it has instead of triggers. (I covered one such surprise in Discover potential obstacles in your design.) Not providing the full text of an error (including error number) - Extended Events is great for capturing those precisely. Misleading information. Example: I have two identical databases, but my query has two different plans. Spoiler alert: schema was the same, but row count and distribution were not. You might not know what info is relevant and which isn't, but try to provide as much context as possible anyway.\n4. Put in some effort Other people want to help you, but they don't want to solve everything for you. The least you could do is:\nPrepare the scripts to reproduce the problem. Practice creating tables and inserting data into them Show your work - what have you tried so far, what problems are you running into. When requesting help with query results show the expected format. Sometimes, it's easier to understand the logic from the requested output than from a text description. Try to find the bare minimum of code to repeat the issue and share that instead. When you have complex queries with tens of joins and hundreds of selected columns - usually they're not all required to debug the problem. The message I'm trying to get across is: Help us to help you. Don't be lazy.\n","permalink":"/posts/dont-be-lazy/","tags":["Debugging","Productivity"],"title":"Don't Be Lazy"},{"categories":["T-SQL Tuesday"],"contents":" T-SQL Tuesday #143 Hosted by John McCormack Topic: Short code examples I've mentioned some of my favourite snippets in my other blog posts. I have plenty more to share, but most of them don't qualify as short. Also, some of them (like the Tally table) are so common that I'm going to skip them.\nTo keep things interesting I've also added a few RegEx scripts, just because I'm a fan of the syntax.\nRemove the square brackets // Find \\[([^]]+)\\] // Replace $1 SQL formatters usually leave square brackets untouched. And I've seen code where everything was wrapped in them.\nYou can try it here along with the explanation.\nReplace table variable declaration with a temp table // Find declare\\s+@([^\\s]+)\\s+(?:as\\s+)?table // Replace CREATE TABLE #$1 Don't forget to add a case-insensitivity flag. Again, you can try it here.\nTime loop Sometimes I need to take snapshots of DMVs for a period of time, or just timebox an operation.\nDECLARE @monitorLoop int = 1 /* init the monitor loop */ DECLARE @startTime datetime2(0) = SYSDATETIME() /* change the datepart and/or value to your needs */ WHILE DATEADD(minute, 60, @startTime) \u003e SYSDATETIME() BEGIN RAISERROR ('Loop number: %i', 1,1, @monitorLoop) WITH NOWAIT /* Do your thing here */ SET @monitorLoop += 1 WAITFOR DELAY '00:01:00' END Impersonation I use this most of the time to test permissions. Warning Careful, you cannot use impersonation to test the module signing permissions. I've learned this the hard way. EXECUTE AS LOGIN = 'MaintenanceNET' /* pick login*/ -- EXECUTE AS USER = 'MaintenanceNET' /* or pick a user */ SELECT SUSER_NAME(), USER_NAME(), ORIGINAL_LOGIN() /* check if you are impersonating */ REVERT /* After you are done, revert to the original login/user */ Recreate an empty database Sometimes when I'm experimenting in a local environment, I just want to delete the database and start from scratch. There might be open connections blocking the DROP. Instead of hunting them down, I use this snippet. Replace the DBName with the name of your DB.\nUSE [master] GO ALTER DATABASE [DBName] SET SINGLE_USER WITH ROLLBACK IMMEDIATE DROP DATABASE [DBName] GO CREATE DATABASE [DBName] Find referencing objects When analysing a database object, I often need to find dependencies. Nothing ever finds 100% of the dependencies, but this code gets me close.\n/* my references */ ;WITH baseReferences AS ( SELECT CONCAT(dsp.referencing_schema_name + '.', dsp.referencing_entity_name) AS referencingObject , CONCAT(dsc.referenced_schema_name + '.', dsc.referenced_entity_name) AS referencedObject , dsc.referenced_minor_name AS referencedColumn , dsc.is_caller_dependent , dsc.is_ambiguous , dsc.is_selected , dsc.is_updated , dsc.is_select_all , dsc.is_all_columns_found , dsc.is_insert_all , dsc.is_incomplete FROM sys.objects AS o CROSS APPLY sys.dm_sql_referencing_entities(CONCAT(SCHEMA_NAME(o.schema_id),'.', o.name), 'object') dsp CROSS APPLY sys.dm_sql_referenced_entities(CONCAT(dsp.referencing_schema_name, '.', dsp.referencing_entity_name), 'object') dsc WHERE o.object_id = OBJECT_ID('PutYourObjectNameHere') AND dsc.referenced_entity_name = o.name ) SELECT DISTINCT PARSENAME(br.referencingObject, 1) --* FROM baseReferences br /* Uncomment to check if the referencing code changes the underlying table */ --WHERE br.is_updated = 1 OR br.is_insert_all = 1 ","permalink":"/posts/short-code-examples/","tags":["Regex","Productivity"],"title":"Short Code Examples (T-SQL Tuesday #143)"},{"categories":["How to"],"contents":"Did you know that when you try to update a VIEW that has an INSTEAD OF TRIGGER, you cannot use the UPDATE statement with a JOIN clause? Probably not. Why would you?\nThat edge case won't appear in the VIEW documentation or the TRIGGER documentation. But when two features collide in SQL Server, the failure mode has to live somewhere - and that place is sys.messages.\nWhenever I'm considering a SQL Server feature I haven't used yet, I run a quick search against the error messages before committing to a design:\nSELECT * FROM sys.messages AS m WHERE m.language_id = 1033 /* English */ AND m.[text] LIKE '%view%instead of%' One of the results:\nUPDATE is not allowed because the statement updates view \"%.*ls\" which participates in a join and has an INSTEAD OF UPDATE trigger. It doesn't work 100% of the time - sometimes the keyword you're looking for is swallowed by a variable placeholder like %.*ls. But it's a useful signal for catching incompatibilities before they catch you.\nAre you considering In-Memory OLTP? Columnstore? Partitioning? Or maybe all of them together? Check the error messages before you commit to a design. It might save you a headache later.\nStill not convinced? Thinking about enabling Full-Text Search?\nSELECT * FROM sys.messages AS m WHERE m.language_id = 1033 AND m.[text] LIKE '%full-text%' Run that first. Now you know what you're getting into.\nIf errors make it to production anyway, Extended Events are how you hunt them down.\n","permalink":"/posts/discover-obstacles/","tags":["Debugging"],"title":"Discover potential obstacles in your design"},{"categories":["Deep Dive"],"contents":"Foreword Before you start tweaking Query Store settings, it helps to know what the defaults actually are. Not the documentation defaults, but what SQL Server actually sets. Turns out those two things don't always match.\nTurn on the Query Store and check the settings Let's create a new database and enable the Query Store.\nWarning Disclaimer: A new database is created from the system model database. If you have changed the Query Store settings there, you will get a different result. USE master CREATE DATABASE QS_Test /* Create a new DB */ GO ALTER DATABASE QS_Test /* Enable the QS */ SET QUERY_STORE = ON (OPERATION_MODE = READ_WRITE) The documentation says we should be getting these values:\nConfiguration Default MAX_STORAGE_SIZE_MB 100 INTERVAL_LENGTH_MINUTES 60 STALE_QUERY_THRESHOLD_DAYS 30 SIZE_BASED_CLEANUP_MODE AUTO QUERY_CAPTURE_MODE AUTO DATA_FLUSH_INTERVAL_SECONDS 900 Warning Disclaimer: This test has been run on SQL Server 2019. On versions 2016 and 2017 the default Capture Mode is Auto. We can check it with this query:\nUSE QS_Test /* Change context to our DB */ SELECT max_storage_size_mb, interval_length_minutes, stale_query_threshold_days, size_based_cleanup_mode_desc, query_capture_mode_desc, flush_interval_seconds FROM sys.database_query_store_options The first value (max_storage_size_mb) doesn't match up. The documentation says 100 but I got 1000.\nAlso what exactly is the value AUTO for query_capture_mode and size_based_cleanup_mode?\nQUERY_CAPTURE_MODE There are 4 options for the Query_Capture_Mode (described in detail here):\nNone All Custom Auto The first two are pretty self-explanatory.\nThe Custom is the one that's interesting. It has four parameters defining the policy.\nThe Custom policy has two parts: a time window and three conditions. STALE_CAPTURE_POLICY_THRESHOLD (default 1 day) defines the evaluation period - think of it as \"the time limit\". During that window, at least one of these three must be true for the query to get saved:\nEXECUTION_COUNT (default 30) TOTAL_COMPILE_CPU_TIME_MS (default 1000) TOTAL_EXECUTION_CPU_TIME_MS (default 100) If we take the Custom policy defaults, our query has to:\nExecute 30 times\nOR total compile CPU time needs to be \u003e 1s\nOR total execution CPU time needs to be \u003e 100ms\nall during 24 hours.\nCapture Mode: Auto Does that mean that the Auto is the same as the Custom defaults?\nTo dig into that, we'll have to use Extended Events. More specifically the query_store_db_settings_and_state event. If XE sessions are new to you, I covered setting one up in this post. A minimalistic session without a target (Watch Live Data) will do.\nCREATE EVENT SESSION [QS_Settings] ON SERVER ADD EVENT qds.query_store_db_settings_and_state GO It runs pretty infrequently (even tens of minutes on my machine) and creates a Query Store settings snapshot for each non-system database.\nWe can see that this time the values match the default of the Custom execution policy.\nWe found two surprises: max_storage_size_mb is 1000, not the 100 the documentation suggests. And Auto capture mode turns out to be just Custom mode with default thresholds. Armed with that, you can make informed decisions before you start tweaking.\n","permalink":"/posts/query-store-default-settings/","tags":["Query Store","Extended Events","Performance"],"title":"Query Store default settings"},{"categories":["How to"],"contents":"Scenario Production is throwing errors. You know they're happening - maybe from a monitoring alert, maybe from an angry email - but you can't reproduce them locally. The call stack is buried in a nested procedure chain and nobody knows which one is failing.\nThis is my go-to technique for these situations. An Extended Events session that captures errors with their full calling stack, plus a script to parse that stack into something readable.\nThe problem Like in a real-world scenario, I'll start with the problem first.\nI will create a procedure SharedLogic that either passes or fails based on the parameter.\nThen I will simulate nesting by creating two more procedures Caller1 and Caller2 that are wrappers around this procedure.\nCREATE OR ALTER PROCEDURE dbo.SharedLogic (@trueFalse bit) AS IF (@trueFalse = 1) /* golden path - no error */ SELECT 1 as Pass ELSE SELECT 1/0 as Error /* Generate a divide by 0 error */ GO CREATE OR ALTER PROCEDURE dbo.Caller1(@passThrough bit) /* Simulate nesting 1 */ AS EXEC dbo.SharedLogic @trueFalse = @passThrough GO CREATE OR ALTER PROCEDURE dbo.Caller2 (@passThrough bit) /* Simulate nesting 2*/ AS EXEC dbo.SharedLogic @trueFalse = @passThrough GO Once the procedures are created, we can test the errors by calling all combinations of parameters.\nGO EXEC dbo.SharedLogic @trueFalse = 0 /* Error */ GO EXEC dbo.SharedLogic @trueFalse = 1 GO EXEC dbo.Caller1 @passThrough = 0 /* Error */ GO EXEC dbo.Caller1 @passThrough = 1 GO EXEC dbo.Caller2 @passThrough = 0 /* Error */ GO EXEC dbo.Caller2 @passThrough = 1 GO Note I'm using the batch separator GO to capture a single statement at a time. Capturing the errors To capture the errors, we'll use an Extended Events session. Here's a basic example.\nCREATE EVENT SESSION [Error_reported] ON SERVER ADD EVENT sqlserver.error_reported ( ACTION ( sqlserver.server_instance_name /* good practice for multi server querying */ , sqlserver.client_app_name /* helps locate the calling app */ , sqlserver.client_hostname /* calling computer name */ , sqlserver.server_principal_name /* can be switched to a user */ , sqlserver.database_id /* can be switched to a database_name */ , sqlserver.sql_text /* grab calling parameters from input buffer */ , sqlserver.tsql_stack /* get the whole stack for parsing later */ ) WHERE ( severity \u003e 10 /* Please test and provide additional filters! */ ) ) ADD TARGET package0.event_file ( SET filename=N'Error_reported' , max_file_size= 20 /* MB */ ) Warning For your system, be sure to test first and filter out benign errors and false positives. Note Starting an Extended Events session requires ALTER ANY EVENT SESSION permission (or CONTROL SERVER). Analysing the errors Let's start the Extended Events session. You can do this with TSQL or through the GUI in Object Explorer.\nALTER EVENT SESSION Error_reported ON SERVER STATE = Start /* Stop */ Don't forget that Extended Events sessions are instance-level objects - that means you are monitoring all databases. If you only want to monitor a specific database, add it to the filters. In our local environment, we can right-click the session and select XE Watch Live Data XE Watch Live Data Once you have an XE session running, you can stream its events live in SSMS without needing a file target. Right-click the session in Object Explorer and choose Watch Live Data.\nCaveats:\nNo memory, no history. There is no target backing the session, so you can only see events that fire while you are actively watching. Anything that happened before you opened the live view is gone. The session keeps running. Closing the Watch Live Data tab does not stop the session. It is still capturing and discarding events in the background - like a tree falling in a forest with no one around to hear it. Stop or drop the session explicitly when you are done. . For production, you should analyse the saved file on disk.\nLet's rerun the EXEC statements to generate the errors once more. We can see that three events popped up. Let's use the Extended Events menu option Choose columns… to pick the columns of interest. I have only chosen a few so they can fit on a screenshot.\nThe XE actions columns (instance, database, app_name, host_name and login) help with locating the calling process. sql_text provides us with an example of how to repro the problem along with the parameter values. And tsql_stack is my favourite. We can query the plan cache and parse the path through the code using the Parse TSQL Stack Parse TSQL Stack Parse the tsql_stack XML from an Extended Events session into a readable call stack. Paste the \u003cframes\u003e element from the XE event data into the @stackOrFrame variable.\nThe COALESCE handles both the old (handle/offsetStart/offsetEnd) and new (sqlhandle/stmtstart/stmtend) XE frame attribute names.\n/* Paste the \u003cframes\u003e\u003c/frames\u003e here */ DECLARE @stackOrFrame xml = '' ;WITH xmlShred AS ( SELECT COALESCE ( CONVERT(varbinary(64), f.n.value('.[1]/@handle', 'varchar(max)'), 1), CONVERT(varbinary(64), f.n.value('.[1]/@sqlhandle', 'varchar(max)'), 1) ) AS handle, COALESCE ( f.n.value('.[1]/@offsetStart', 'int'), f.n.value('.[1]/@stmtstart', 'int') ) AS offsetStart, COALESCE ( f.n.value('.[1]/@offsetEnd', 'int'), f.n.value('.[1]/@stmtend', 'int') ) AS offsetEnd, f.n.value('.[1]/@line', 'int') AS line, f.n.value('.[1]/@level', 'tinyint') AS stackLevel FROM @stackOrFrame.nodes('//frame') AS f(n) ) SELECT xs.stackLevel, ca.outerText, ca2.statementText FROM xmlShred AS xs CROSS APPLY sys.dm_exec_sql_text(xs.handle) AS dest CROSS APPLY (SELECT LTRIM(RTRIM(dest.text)) FOR XML PATH(''), TYPE) AS ca(outerText) CROSS APPLY ( SELECT SUBSTRING ( dest.text, (xs.offsetStart / 2) + 1, (( CASE WHEN xs.offsetEnd = -1 THEN DATALENGTH(dest.text) ELSE xs.offsetEnd END - xs.offsetStart ) / 2) + 1 ) FOR XML PATH(''), TYPE ) AS ca2(statementText) ORDER BY xs.stackLevel OPTION (RECOMPILE); I have CAST the text to XML so it's formatted nicely, but if your code contains XML-specific special characters, it might break.\nWarning The stack parsing relies on sys.dm_exec_sql_text, which reads from the plan cache. If the plan has been evicted (server restart, memory pressure, DBCC FREEPROCCACHE), the query returns NULL. Run it while the plans are still cached. . Just paste the stack XML into the @stackOrFrame variable, and we should get the following output.\nConclusion Set up the session, let it run, and check back when errors happen. The stack parsing query is the real payoff - instead of guessing which procedure in a 6-level deep chain is the culprit, you get the exact path.\nStart with the basic session from this post and adapt the filters to your environment. The most common mistake is not filtering enough - without additional predicates, you'll drown in noise from benign errors that fire constantly.\n","permalink":"/posts/investigating-errors-with-extended-events/","tags":["Extended Events","XML","Debugging"],"title":"Investigating Errors With Extended Events"},{"categories":["Non-technical"],"contents":"Discontent with the WordPress A long time ago when I started with this blog, there was no real plan. I read that blogs are usually done in WordPress and rolled with it.\nBut over time I was so unhappy that even the mere thought of writing about something I enjoy was immediately dampened by the struggles I had.\nThe never-ending updates to plugins, the theme I chose but didn't really understand and especially my troubles with images - I was just overwhelmed.\nStatic Generated sites to the rescue As you might have guessed, this isn't an area where I am well-versed. My first run-in with static sites was when Chad Baldwin mentioned on the SQL Server Slack channel having a blog on GitHub Pages using Jekyll. I didn't pay attention then - it also seemed like a lot of work.\nEnter Hugo But then later both Kendra Little wrote about Moving from WordPress to Hugo and Cathrine Wilhelmsen wrote about Goodbye WordPress, Hello Hugo wrote about it as well, it grabbed my attention. Especially the choice of Hugo - being able to build a site with simple Markdown and preview it almost instantly. Then you can use an automatic pipeline to publish it (to Azure Static Website for example).\nI'm not gonna go into details about how to set it all up because I've used Kendra's amazing article (in fact so much, that it feels like stealing). I can also recommend Justin Bird wrote about Hugo series and Mike Dane's tutorials on YouTube.\nThe one recommendation I can give? Find a page on your favourite tech person's blog that has images, code samples, etc. and try to copy it on your blog. That way you can see how it would compare.\nIt's alive The Markdown and code-first approach to blogging resonated with me so much, that I switched from WordPress even though this site is still very much a work in progress.\nDon't let \"perfect\" be the enemy of \"good\". — attributed to Voltaire There is still a lot of work ahead, but this time I'm optimistic. I'm tracking planned features and post ideas on the GitHub project board.\n","permalink":"/posts/out-with-the-old/","tags":["Personal"],"title":"Out with the old…"},{"categories":["T-SQL Tuesday"],"contents":" T-SQL Tuesday #141 Hosted by TJay Belt Topic: Work-life balance I spent a month tracking every hour of my day. I expected the data to confirm that I was overworked and had no time for anything. It didn't.\nHere's the graph. Turns out things weren't nearly as bad as I made them out to be.\nSleep takes up about a third of each day (no surprise). Work is a consistent block but not the monster I imagined. Family time, learning, and my own time all have their share. The problem wasn't the amount of free time - it was how I felt about it.\nWhile my job is pretty great and I clock off exactly after 8 hours, it doesn't stop there. It's still in the back of my mind.\nAnd SQL is not just my job, it is my hobby as well. I wanted to learn in my free time, experiment, maybe blog about it. But when anything got in the way, I got frustrated. These are my lessons learned that will hopefully help someone else as well.\n1. Manage your expectations I think this is the most important one. I've set too many goals for myself, wanted to overachieve. When anything unplanned happened I fell short of my target and blamed myself. Admit some things are beyond your control and you won't feel as much pressure.\n2. Get enough sleep Sleep is the main way of recharging. If you don't start your day well-rested, it will affect you. I found having a routine helped a lot, but I struggle to keep at it.\nAs for sleep, maybe try out a white noise generator or dark blinds, a new pillow or mattress - experiment and find out what works for you. It's also good to limit blue light exposure before sleep. I haven't sacrificed my computer/phone time but instead installed the yellow tint which should be easier on the eyes.\n3. Take some time off Some people like watching sports, some like television or games, some scroll endlessly through social media - the point is everyone is different. Allow yourself some rest. Otherwise, you will burn out.\nThe time you enjoy wasting is not wasted. — Bertrand Russell (probably) Looking back at that graph, my time allocation wasn't the real problem. My expectations were. Once I stopped treating every evening as an opportunity to be \"productive\", things got easier.\n","permalink":"/posts/work-life-balance/","tags":["Personal"],"title":"Work Life Balance (T-SQL Tuesday #141)"},{"categories":["T-SQL Tuesday"],"contents":" T-SQL Tuesday #140 Hosted by Anthony Nocentino Topic: Containers It took me three tries before Docker finally clicked.\nMy experience with containers (Docker) can be summed up by this quote:\nTell me and I forget.\nTeach me and I remember.\nInvolve me and I learn.\nThe first time I just needed a dev environment for RabbitMQ. I followed a guide. I didn't understand everything I did, but it worked out well, and I've liked the \"cleanness\" of containers - leaving nothing behind when I was done.\nThe second time I wanted to roll out open source monitoring using a guide from Tracy Boggiano . All I needed was Grafana which I didn't have locally. Docker to the rescue. Now I was aware of Docker and it was always in the back of my mind.\nThe third time (Was a charm) I found a use case for a project of my own. I needed an easy to set-up environment for my build pipeline. This time I finally learned a lot - not just about Docker, but SQL on Linux, PowerShell, etc.\nI encourage everyone to find a use case for containers in their projects because it genuinely is the best way to learn. I can recommend the SQL Server and Containers guide from Andrew Pruski (DBA From The Cold) .\n","permalink":"/posts/containers-and-me/","tags":["Docker"],"title":"Containers and Me (T-SQL Tuesday #140)"},{"categories":["T-SQL Tuesday"],"contents":" T-SQL Tuesday #136 Hosted by Brent Ozar Topic: Favorite data type I'm mostly indifferent to data types - it's like asking what is your favorite spatula. They each have their purpose. I admit I like legacy data types because they provide job security but they are not really my favourite.\nAfter a bit of thinking, I'll have to go with the… *drumroll*… sql_variant.\nHuh, sql_variant? But why? I've met DB developers who weren't even aware of its existence.\nFirst of all, it can hold the majority of other data types, so it's superior by definition. It's at the top of the data precedence list of the system data types. It's got a cool underscore in its name. No implicit conversion errors when converting from the sql_variant, because you have to use explicit conversion (hey, it's a feature!).\nBut the real reason is the SQL_VARIANT_PROPERTY function. With that, I can get\nBaseType Collation MaxLength Precision Scale TotalBytes of different data types. I had an indexed view that multiplied two decimal numbers - what was the resulting size of that column? We can use sql_variant to figure this out.\nDECLARE @number1 decimal (8,4) = 1234.4321 , @number2 decimal (14,6) = 12345678.654321 , @myFavourite sql_variant SELECT @myFavourite = @number1 * @number2 SELECT SQL_VARIANT_PROPERTY(@myFavourite, 'BaseType') AS BaseType , SQL_VARIANT_PROPERTY(@myFavourite, 'Collation') AS [Collation] , SQL_VARIANT_PROPERTY(@myFavourite, 'MaxLength') AS MaxLength , SQL_VARIANT_PROPERTY(@myFavourite, 'Precision') AS [Precision] , SQL_VARIANT_PROPERTY(@myFavourite, 'Scale') AS Scale , SQL_VARIANT_PROPERTY(@myFavourite, 'TotalBytes') AS TotalBytes I'll leave the result as an exercise for the reader.\nI've seen it used only once in production in an EAV table where the sql_variant column was holding the values and a different column described the required data type. But maybe that's exactly why I like it - because it's so rare.\nFor the least favourite type - datetime, hands down. It's everywhere (even in non-legacy databases) even though it's replaced by the superior datetime2. There is some millisecond rounding and it can eat up more bytes than a more specific date type - but none of this is serious enough that it warrants a rewrite.\nSo it's this minor inefficiency that will always haunt the database and that's why I hate it.\n","permalink":"/posts/my-favourite-data-type/","tags":null,"title":"My Favourite Data Type (T-SQL Tuesday #136)"},{"categories":["Non-technical"],"contents":"Small kids, a home office that doubles as a storage area, and neighbours who love to drill into shared walls at random hours. The perfect setting for an exam where unexpected sounds can get you disqualified.\nBackground With the MCSA exams being retired at the end of January and COVID restrictions ramping up in my country causing the test centres to be closed, I had no other option but to go online.\nI was worried about this as I heard stories from my colleagues that the slightest unexpected sound might disqualify you. So I looked up a guide on how it all goes down. This one was the most informative.\nMy family had to leave for the duration of the exam. Not nice to kick them out in the winter, but they took one for the team. I was also hoping no one would ring a bell, start to drill the shared walls, or randomly scream below my window (which are all frequent events).\nWith that in mind, I got to it. First, you download and run the system test program. This is the same program that will be used on the day of the real exam. You will get an Access code for the test and later a different one for the exam.\nIt will check your microphone, internet, and webcam feed. Next, you will need to close all applications and processes you possibly can. Anecdotally, I once forgot to close an app and when I got to the \"Launch simulation\" it told me that one app hasn't been closed and this incident has been logged.\nPrerequisites Computer (duh) preferably desktop. They specifically warn about VPNs, proxies, and company computers with firewall rules, etc. One valid government-issued ID (in my case driver's license) Mobile phone for verification (selfie, photo of ID, photos of the room, point of contact in case of e.g. internet outage). The mobile phone was my biggest point of confusion. You cannot turn it off, because the proctor might contact you, but you also cannot have any random sounds? How am I supposed to control who calls me? I had the phone outside of my reach and screen down as per guidelines, but it was buzzing constantly with messages from company chat. Webcam and microphone A closed room where no one will disturb you A clean work area: no papers, pens, jewellery, other screens, etc. Nothing that could be used to cheat or copy the questions. I have two monitors on my stand, so I had to turn the secondary away from me and unplug it. Also, there was a mess in the rest of the room as my \"office\" is sort of a storage area. On the day of the exam You can start the process between 30 minutes before and 15 minutes after the registered time. Go to the bathroom in advance, you cannot during the exam.\nYou will go to your certification dashboard and it will show in the Appointments section. You will get the real access code this time and start the verification process. You will be prompted to use a mobile phone and go to the website and take a selfie, a photo of your Id against a dark background with no glare (both sides), and then 4 photos of your work area. Facing the desk, behind the desk, left and right.\nI recommend starting as early as possible because I had some technical difficulties. First, my primary mobile browser is Firefox, and I couldn't get past the selfie stage. It just failed silently. Only after switching to Chrome did the selfie camera show an overlay with an oval area where my face should be.\nThe second problem was with taking photos of my room. I took the first one and on the second it said: internal memory error. I tried three times with no luck, so I restarted the phone and changed the camera app from custom to default. That did the trick.\nAfter completing verification, accepting the terms and conditions, and all that jazz, it went straight to the exam. I could see my camera feed at the top of the screen. Your face must not move outside the camera scope, otherwise you will be disqualified. Also, there is a Whiteboard option (similar to the piece of paper you would get at the test centre) and a chat with the proctor.\nI was expecting the greeter to give me some basic info and also slowly move my web camera from left to right to show my work area (like the official video said), but nothing happened.\nFinal thoughts The test itself was not that much different from the test centre. But the feeling of being watched the whole time made me super itchy, and I had the urge to scratch all the time. I was also mindful of not putting my hands over my mouth when thinking or mumbling to myself, as these might be mistaken for a cheating attempt.\nOverall I was more worried about failing for things outside my control (like the mobile phone or noise) than my lack of knowledge. Given a choice, I would pick the test centre next time, even though that means a 2-hour commute one-way for me.\n","permalink":"/posts/my-experience-doing-the-online-proctored-exam/","tags":["Personal","Certification"],"title":"My Experience Doing the Online Proctored Exam"},{"categories":["How to"],"contents":"Foreword \"It depends\" is the DBA's most frequent answer, and for a good reason. Most of the time giving good advice really depends on many variables. A good DBA must follow up by explaining \"why\" it depends.\nAnother often-repeated mantra is \"There are no silver bullets.\" Everything is a trade-off. Parallelism might decrease query duration, for example, but it increases CPU usage.\nWell, this blog post (and hopefully series) will try to cover a few of those very rare silver bullets. Now I'm cheating a bit in my definition, these tips will be along the lines \"did you know this existed?\" and the alternative is writing the query in a sub-optimal way. Let's get to it then!\nTYPE directive in FOR XML XML is a very polarizing topic in the SQL community, some developers might be lucky and never run into it. But love it or hate it, it is there and will be for a while so we might as well use it in the most efficient manner.\nLet's say our goal is to generate an XML document from a query and pass it to an application or Service Broker, etc. We're going to use the FOR XML query. I'm testing this in a fresh SQL Server 2019 instance in the master database.\nI'm going to generate a small XML document into an xml variable both with and without the TYPE directive. I'll repeat the experiment but save it into the varchar(MAX) and nvarchar(MAX) variables.\n/* xml variable, no TYPE directive */ DECLARE @xmlNoType xml SET @xmlNoType = ( SELECT * FROM sys.all_objects FOR XML PATH(''), ROOT ('Document') ) SELECT (DATALENGTH(@xmlNoType) / 1024.0) / 1024. AS SizeMB GO /* xml variable, TYPE directive */ DECLARE @xmlType xml SET @xmlType = ( SELECT * FROM sys.all_objects FOR XML PATH(''), ROOT ('Document'), TYPE ) --SELECT (DATALENGTH(@xmlType) / 1024.0) / 1024. AS SizeMB GO /* varchar(MAX) variable, no TYPE directive */ DECLARE @varchar varchar(MAX) SET @varchar = ( SELECT * FROM sys.all_objects FOR XML PATH(''), ROOT ('Document') /* Cannot use TYPE: Implicit conversion from data type xml to varchar(max) is not allowed. Use the CONVERT function to run this query. */ ) SELECT (DATALENGTH(@varchar) / 1024.0) / 1024. AS SizeMB GO /* nvarchar(MAX) variable, no TYPE directive */ DECLARE @nvarchar nvarchar(MAX) SET @nvarchar = ( SELECT * FROM sys.all_objects FOR XML PATH(''), ROOT ('Document') /* Cannot use TYPE: Implicit conversion from data type xml to nvarchar(max) is not allowed. Use the CONVERT function to run this query. */ ) SELECT (DATALENGTH(@nvarchar) / 1024.0) / 1024. AS SizeMB GO /* XML size 0.59 MB varchar(max) size 0.90 MB nvarchar(max) size 1.80 MB */ Results on my machine are the following: first the size. Unsurprisingly both with and without TYPE directive, the XML size is the same - roughly 0.59 MB varchar(MAX) variable is 0.9 MB and nvarchar(MAX) is double that at 1.8 MB.\nLet's look at the CPU and elapsed time.\nUsing TYPE saved about 6k reads and cut both CPU and duration to roughly half compared to the non-TYPE version. The non-xml variables aren't bad performance-wise either, but they come with more reads and a larger data type footprint.\nLet's compare the actual execution plan:\nIn the query without the TYPE there is an implicit conversion. The TYPE directive tells SQL Server you want to generate an XML document. Without that, it generates an nvarchar(MAX) and then implicitly casts it to XML.\nDECLARE @whatIsThis SQL_VARIANT SET @whatIsThis = ( SELECT * FROM sys.all_objects FOR XML PATH(''), ROOT ('Document') ) That returns an error message:\nMsg 206, Level 16, State 2, Line 5\nOperand type clash: nvarchar(max) is incompatible with sql_variant I've repeated the tests but doing a cartesian product with several values to enlarge the XML. These are the results for 6 MB and 21 MB XML respectively. The varchar(max) size was 20 MB and 70 MB (nvarchar is doubled).\nWith larger documents, the benefits only grow for such a small change. It earns its spot in the Silver bullets series. I've run into processes that generated XMLs over a hundred MBs and the savings were vast.\n","permalink":"/posts/generate-xml-documents-efficiently/","tags":["XML","Performance"],"title":"Generate XML documents efficiently"}]