May 31 2016

Precedence Goes to Query Store or Plan Guide?

While presenting at SQLDay in Wroclaw, Poland, on the Query Store, I was asked a pretty simple question, which takes precedence, the Query Store or a Plan Guide?

One of my favorite answers to questions is “I don’t know” because it gives me the opportunity to learn. Let’s figure this one out together.

I’ll post the code to recreate this experiment within AdventureWorks at the end of the article. I’m doing this because the code for forcing execution plans using Plan Guides can be pretty doggone long (you may need to generate your own XML from a plan on your own system, fair warning).

I have a stored procedure that I use frequently to demonstrate parameter sniffing and bad parameter sniffing, AddressByCity (listed below). This query when passed the value ‘London’ will generate a plan with a Merge Join. When passed the value ‘Mentor’ it will generate a plan with a Loops Join. These different plans are caused by the statistics for the different values. 434 rows for ‘London’ and 1 row for ‘Mentor’. To start the test, I’ll generate a plan for each value, dropping the plan from cache between executions, so that I have both plans in the Query Store. Now, we’re ready for phase 1.

Phase 1

I’m going to remove the current plan from cache. I will then use sp_query_store_force_plan to select the plan with the Loops join. I’ll run the query using ‘London’ to validate that the plan generated is NOT the plan for ‘London’ but rather the plan for ‘Mentor’. Done, and I can see the Use Plan results in the Properties of the Select operator:

QSvPG_UsePlan

I’ve highlighted a couple of interesting points. First, even though we are using a plan that is compiled for the value of ‘Mentor’, because this plan was compiled using the value ‘London’, that’s what gets stored, despite the plan used. You can then see down at the bottom, that the ‘Use plan’ property is set to true. Now, let’s force the plan using a plan guide (in my script below). To be very sure what happens, I’ll again clear the plan from cache.

If the Query Store wins, I should see a plan with a Loops Join. If the Plan Guide wins, I should see a plan with a Merge Join. The moment of truth:

QSvPG_Plan

Query Store wins! Let’s see how it shows in the Properties:

QSvPG_PlanGuideName

You’ll see that the Parameter Compile & Runtime values are the same and that the ‘Use plan’ property is the same. Note however, the PlanGuideName property is in play. In fact, the indication is that the optimizer thinks that a Plan Guide has been successfully applied. In fact, if I watch the Extended Events for the plan_guide_successful event, the best way to check how Plan Guides are being used, apart from looking at the execution plans, it shows success for the Plan Guide we created.

Phase 2

Precedence could matter. First, I remove the Plan Guide using sp_control_plan_guide. Then, I remove the Query Store force using sp_query_store_unforce_plan. I’ll also clear the plan out of cache again, just to be sure. Next, I create the Plan Guide. I’ll run the procedure using the value ‘Mentor’ to ensure that I see the Merge/’London’ plan, the PlanGuideName property in that plan, and a successful application of a Plan Guide from the plan_guide_successful ExEvent. All good.

Now, I once again force the plan from Query Store. Rerun the plan…

The winner is Query Store.

Phase 3

OK, so, if we try to force plans using Plan Guide, Query Store wins. What about if I’m just trying to apply a hint? Let’s clear everything up again and then create an OPTIMIZE FOR query hint in a Plan Guide. Simple right?

Query Store wins again… Except things get a little odd. Take a look at the Properties now:

QSvPG_Parameter

Normally, with a Plan Guide that includes an OPTIMIZE FOR hint, I’d see the Compiled Value as the defined value within the hint (in this case ‘London’). However, this time I’m seeing just the value of ‘Mentor’.

Conclusion

I don’t know what’s occurring in the internals. I just know the evidence I have in front of me. First, Query Store vs. Plan Guides? Query Store wins. However, internally, something is still going on. The Plan Guide is still successfully applied, at least according to what I can see. The processes that determine if a Plan Guide should be applied or not are not interfered with by the Query Store plan forcing. The fact that I’m seeing a Query Store plan and evidence of a successful Plan Guide though, that’s a problem that could cause issues when troubleshooting. Now you know, and knowing, as they say, is half the battle.


Let’s get together and talk about execution plans. In August, I’ll be doing an all day pre-con at SQLServer Geeks Annual Summit in Bangalore India.

I’m extremely exited to be in Oslo Norway for a pre-con before SQL Saturday Oslo in September.


/*NOTE: These scripts are not in the exact order in which
they must be run to re-create all my tests. These are
however, all the scripts you'll need.
*/
--the procedure
CREATE PROC dbo.AddressByCity @City NVARCHAR(30)
AS
SELECT  a.AddressID,
        a.AddressLine1,
        a.AddressLine2,
        a.City,
        sp.Name AS StateProvinceName,
        a.PostalCode
FROM    Person.Address AS a
JOIN    Person.StateProvince AS sp
        ON a.StateProvinceID = sp.StateProvinceID
WHERE   a.City = @City;
GO

--generating the first plan
EXEC dbo.AddressByCity
    @City = N'London';



--remove this from cache
DECLARE @PlanHandle VARBINARY(64);

SELECT  @PlanHandle = deps.plan_handle
FROM    sys.dm_exec_procedure_stats AS deps
WHERE   deps.object_id = OBJECT_ID('dbo.AddressByCity');

IF @PlanHandle IS NOT NULL
    BEGIN
        DBCC FREEPROCCACHE(@PlanHandle);
    END
GO



--generate the second plan
EXEC dbo.AddressByCity
    @City = N'Mentor';




--get the query & plan id
SELECT  qsq.query_id,
        qsp.plan_id
FROM    sys.query_store_query AS qsq
JOIN    sys.query_store_plan AS qsp
        ON qsp.query_id = qsq.query_id
WHERE   qsq.object_id = OBJECT_ID('dbo.AddressByCity');


--force the plan
EXEC sys.sp_query_store_force_plan 414,460;

--unforce the plan
EXEC sys.sp_query_store_unforce_plan 414,460;




--Get the plan_handle and offset 
SELECT  deqs.plan_handle,
        deqs.statement_start_offset
FROM    sys.dm_exec_query_stats AS deqs
CROSS APPLY sys.dm_exec_sql_text(deqs.sql_handle) AS dest
WHERE   dest.objectid = OBJECT_ID('dbo.AddressByCity');

--create a plan guide
EXEC sys.sp_create_plan_guide_from_handle
    @name = N'PlanGuidePrecedence',
    @plan_handle = 0x05000500C26A7610A03A4F53A600000001000000000000000000000000000000000000000000000000000000,
    @statement_start_offset = 108;

--remove plan guide
EXEC sys.sp_control_plan_guide
    @operation = N'DROP',
    @name = N'PlanGuidePrecedence';


--pull a plan out of cache. You'll need to make sure the right plan is there first
DECLARE @xmlplan NVARCHAR(MAX)

SELECT  @xmlplan = deqp.query_plan
FROM    sys.dm_exec_query_stats AS deqs
CROSS APPLY sys.dm_exec_text_query_plan(deqs.plan_handle,DEFAULT,DEFAULT) AS deqp
WHERE   deqp.objectid = OBJECT_ID('dbo.AddressByCity');

SELECT @xmlplan


--copied and pasted the value from @xmlplan for easy reuse
DECLARE @xmlplan NVARCHAR(MAX) =
'<ShowPlanXML xmlns="http://schemas.microsoft.com/sqlserver/2004/07/showplan" Version="1.5" Build="13.0.1300.275"><BatchSequence><Batch><Statements><StmtSimple StatementText="CREATE PROC dbo.AddressByCity @City NVARCHAR(30)
AS
SELECT  a.AddressID,
        a.AddressLine1,
        a.AddressLine2,
        a.City,
        sp.Name AS StateProvinceName,
        a.PostalCode
FROM    Person.Address AS a
JOIN    Person.StateProvince AS sp
        ON a.StateProvinceID = sp.StateProvinceID
WHERE   a.City = @City" StatementId="1" StatementCompId="3" StatementType="SELECT" StatementSqlHandle="0x0900D56A5A43E8B56B541DFF8DC13F149C6A0000000000000000000000000000000000000000000000000000" DatabaseContextSettingsId="1" ParentObjectId="276196034" StatementParameterizationType="0" RetrievedFromCache="true" StatementSubTreeCost="0.316799" StatementEstRows="434" SecurityPolicyApplied="false" StatementOptmLevel="FULL" QueryHash="0xDD75E124763781F2" QueryPlanHash="0x6C408B64C5E8A83F" StatementOptmEarlyAbortReason="GoodEnoughPlanFound" CardinalityEstimationModelVersion="120"><StatementSetOptions QUOTED_IDENTIFIER="true" ARITHABORT="true" CONCAT_NULL_YIELDS_NULL="true" ANSI_NULLS="true" ANSI_PADDING="true" ANSI_WARNINGS="true" NUMERIC_ROUNDABORT="false"></StatementSetOptions><QueryPlan CachedPlanSize="32" CompileTime="3" CompileCPU="3" CompileMemory="360"><MissingIndexes><MissingIndexGroup Impact="89.7983"><MissingIndex Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]"><ColumnGroup Usage="EQUALITY"><Column Name="[City]" ColumnId="4"></Column></ColumnGroup></MissingIndex></MissingIndexGroup></MissingIndexes><MemoryGrantInfo SerialRequiredMemory="512" SerialDesiredMemory="720"></MemoryGrantInfo><OptimizerHardwareDependentProperties EstimatedAvailableMemoryGrant="260900" EstimatedPagesCached="65225" EstimatedAvailableDegreeOfParallelism="2"></OptimizerHardwareDependentProperties><RelOp NodeId="0" PhysicalOp="Merge Join" LogicalOp="Inner Join" EstimateRows="434" EstimateIO="0" EstimateCPU="0.0069349" AvgRowSize="225" EstimatedTotalSubtreeCost="0.316799" Parallel="0" EstimateRebinds="0" EstimateRewinds="0" EstimatedExecutionMode="Row"><OutputList><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="AddressID"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="AddressLine1"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="AddressLine2"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="City"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="PostalCode"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[StateProvince]" Alias="[sp]" Column="Name"></ColumnReference></OutputList><Merge ManyToMany="0"><InnerSideJoinColumns><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="StateProvinceID"></ColumnReference></InnerSideJoinColumns><OuterSideJoinColumns><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[StateProvince]" Alias="[sp]" Column="StateProvinceID"></ColumnReference></OuterSideJoinColumns><Residual><ScalarOperator ScalarString="[AdventureWorks2014].[Person].[StateProvince].[StateProvinceID] as [sp].[StateProvinceID]=[AdventureWorks2014].[Person].[Address].[StateProvinceID] as [a].[StateProvinceID]"><Compare CompareOp="EQ"><ScalarOperator><Identifier><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[StateProvince]" Alias="[sp]" Column="StateProvinceID"></ColumnReference></Identifier></ScalarOperator><ScalarOperator><Identifier><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="StateProvinceID"></ColumnReference></Identifier></ScalarOperator></Compare></ScalarOperator></Residual><RelOp NodeId="1" PhysicalOp="Clustered Index Scan" LogicalOp="Clustered Index Scan" EstimateRows="181" EstimateIO="0.00386574" EstimateCPU="0.0003561" AvgRowSize="65" EstimatedTotalSubtreeCost="0.00422184" TableCardinality="181" Parallel="0" EstimateRebinds="0" EstimateRewinds="0" EstimatedExecutionMode="Row"><OutputList><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[StateProvince]" Alias="[sp]" Column="StateProvinceID"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[StateProvince]" Alias="[sp]" Column="Name"></ColumnReference></OutputList><IndexScan Ordered="1" ScanDirection="FORWARD" ForcedIndex="0" ForceSeek="0" ForceScan="0" NoExpandHint="0" Storage="RowStore"><DefinedValues><DefinedValue><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[StateProvince]" Alias="[sp]" Column="StateProvinceID"></ColumnReference></DefinedValue><DefinedValue><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[StateProvince]" Alias="[sp]" Column="Name"></ColumnReference></DefinedValue></DefinedValues><Object Database="[AdventureWorks2014]" Schema="[Person]" Table="[StateProvince]" Index="[PK_StateProvince_StateProvinceID]" Alias="[sp]" IndexKind="Clustered" Storage="RowStore"></Object></IndexScan></RelOp><RelOp NodeId="2" PhysicalOp="Sort" LogicalOp="Sort" EstimateRows="434" EstimateIO="0.0112613" EstimateCPU="0.0060321" AvgRowSize="177" EstimatedTotalSubtreeCost="0.30564" Parallel="0" EstimateRebinds="0" EstimateRewinds="0" EstimatedExecutionMode="Row"><OutputList><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="AddressID"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="AddressLine1"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="AddressLine2"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="City"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="StateProvinceID"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="PostalCode"></ColumnReference></OutputList><MemoryFractions Input="1" Output="1"></MemoryFractions><Sort Distinct="0"><OrderBy><OrderByColumn Ascending="1"><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="StateProvinceID"></ColumnReference></OrderByColumn></OrderBy><RelOp NodeId="3" PhysicalOp="Clustered Index Scan" LogicalOp="Clustered Index Scan" EstimateRows="434" EstimateIO="0.257199" EstimateCPU="0.0217324" AvgRowSize="177" EstimatedTotalSubtreeCost="0.278931" TableCardinality="19614" Parallel="0" EstimateRebinds="0" EstimateRewinds="0" EstimatedExecutionMode="Row"><OutputList><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="AddressID"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="AddressLine1"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="AddressLine2"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="City"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="StateProvinceID"></ColumnReference><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="PostalCode"></ColumnReference></OutputList><IndexScan Ordered="0" ForcedIndex="0" ForceScan="0" NoExpandHint="0" Storage="RowStore"><DefinedValues><DefinedValue><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="AddressID"></ColumnReference></DefinedValue><DefinedValue><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="AddressLine1"></ColumnReference></DefinedValue><DefinedValue><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="AddressLine2"></ColumnReference></DefinedValue><DefinedValue><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="City"></ColumnReference></DefinedValue><DefinedValue><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="StateProvinceID"></ColumnReference></DefinedValue><DefinedValue><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="PostalCode"></ColumnReference></DefinedValue></DefinedValues><Object Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Index="[PK_Address_AddressID]" Alias="[a]" IndexKind="Clustered" Storage="RowStore"></Object><Predicate><ScalarOperator ScalarString="[AdventureWorks2014].[Person].[Address].[City] as [a].[City]=[@City]"><Compare CompareOp="EQ"><ScalarOperator><Identifier><ColumnReference Database="[AdventureWorks2014]" Schema="[Person]" Table="[Address]" Alias="[a]" Column="City"></ColumnReference></Identifier></ScalarOperator><ScalarOperator><Identifier><ColumnReference Column="@City"></ColumnReference></Identifier></ScalarOperator></Compare></ScalarOperator></Predicate></IndexScan></RelOp></Sort></RelOp></Merge></RelOp><ParameterList><ColumnReference Column="@City" ParameterCompiledValue="N'London'"></ColumnReference></ParameterList></QueryPlan></StmtSimple></Statements></Batch></BatchSequence></ShowPlanXML>'


EXEC sys.sp_create_plan_guide
    @name = N'PlanGuidePrecedence',
    @stmt = N'SELECT  a.AddressID,
        a.AddressLine1,
        a.AddressLine2,
        a.City,
        sp.Name AS StateProvinceName,
        a.PostalCode
FROM    Person.Address AS a
JOIN    Person.StateProvince AS sp
        ON a.StateProvinceID = sp.StateProvinceID
WHERE   a.City = @City;',
    @type = N'OBJECT',
    @module_or_batch = N'dbo.AddressByCity',
    @params = NULL,
    @hints = @xmlplan;


--create a plan guide with just a hint in it
EXEC sys.sp_create_plan_guide @name = 'PlanGuidePrecedence', -- sysname
    @stmt = N'SELECT  a.AddressID,
        a.AddressLine1,
        a.AddressLine2,
        a.City,
        sp.Name AS StateProvinceName,
        a.PostalCode
FROM    Person.Address AS a
JOIN    Person.StateProvince AS sp
        ON a.StateProvinceID = sp.StateProvinceID
WHERE   a.City = @City;', -- nvarchar(max)
    @type = N'Object', -- nvarchar(60)
    @module_or_batch = N'dbo.AddressByCity', -- nvarchar(max)
    @params = NULL, -- nvarchar(max)
    @hints = N'OPTION(OPTIMIZE FOR(@City = ''London''))'; -- nvarchar(max)
May 25 2016

A Moment For Reflection…

While this is a personal blog, I try to keep it focused on either technical topics or personal development and leadership related to technical topics. This post is a complete deviation from all of the above. Please, read it anyway.

I went to Auschwitz and Birkenau.

Words are failing me here.

Before I went, I was terribly conflicted about the trip. I had the opportunity to visit another Nazi concentration camp, Dachau, and I deferred. I know horror was committed there. Why should I voluntarily subject myself to it? As part of a trip to Poland, Aaron Bertrand suggested we should take a couple of extra days to see the sights, including visiting Auschwitz. I didn’t want to go. Then I started thinking.

Obligation. Honor. Remembrance. Humility. Respect.

I truly don’t consider myself to be all that good a person. However, I’m trying. I see how my betters behave and I try to appropriately ape that behavior. In this case Aaron was setting the path for me, so I followed.

What to say about the place?

The scale is staggering. You may have looked at pictures. You might have seen films. While a picture may paint a thousand words, you need millions of words to appropriately describe the utter enormity of the staggering horror that place represents. 1.3 million people in the door and 1.1 million died. That required massive, focused, efficiency, planning, thought and nearly limitless evil. It’s not the act of foaming at the mouth madmen. Standing in the middle of Birkenau, in the rail yard (a ******* rail yard for ******* humans, by all the gods… anyway…), you can feel and see all that, despite the fact that the place is mostly a ruin. Lots and lots of people visit Auschwitz. The name is synonymous with… all of it. However, fewer go over to Birkenau. Honestly, you have to go to Birkenau. Nothing can prepare you and nothing is equivalent.

Emotionally…

Horror, of course. Way too much rage. Sadness. And, frankly, and very surprisingly, a certain deadness inside. I actually got to the point where I read another plaque about X number of people being processed and there was nothing there. It had all been hollowed out. I think that was the single most surprising moment. My empathy pool and just been utterly drained. It only took 1.3 million people. Of course, with a little time between me and that place, I’m back to tearing up as I type this.

I’m not completely sure why I feel so compelled to share this, yet I must. I cannot recommend enough that you go and visit this place if you have the opportunity. If you don’t have the opportunity, you need to make it. I read  a lot of history, so the concept of “Never Again” is a sad joke. However, maybe, just maybe, if enough people stand in that rail yard, or in front of Incinerator #5 (which means there is 1-4…. rage again), maybe if enough people get the education that place provides, maybe “Never Again” will actually come to be more than just a vain hope.

 

May 24 2016

On the Buckeye Blitz!

In just a few weeks I’ll be doing the Buckeye Blitz. That is, a tour of user groups in Ohio (aka, the Buckeye state, after a tree, not a sports team), one per day across a week. Here’s how it breaks down:

Cleveland: 6/13
Toledo: 6/14
Columbus: 6/15
Cincinnati: 6/16

I’ll be talking on this topic at each of the groups:

Change Your Habits: Tips to Tune Your T-SQL
T-SQL proides many different ways to accomplish the same task, and as you might expect, some ways are better than others. In this session, you will learn specific techniques, that when followed make you a better T-SQL developer. The session is jam-packed with practical examples and is designed for administrators and developers who want to bring their T-SQL skills to the next level. You’ll write clearer and easier to read T-SQL as well as write better performing T-SQL. So useful, you can implement these tips immediately!

I hope you’ll come out and see me on the Buckeye Blitz. I’m really looking forward to it. Plus, I haven’t driven across Ohio since my first day on the job at Redgate. I’ll assume less snow in June.

May 23 2016

PASS Board 2016: Update #3

I didn’t post an update last month, but I did do something more important: I elicited your feedback on a question the Board needs to act on.

Here’s what I’ve been up to.

First, based on your feedback (thanks) and conversations that I’ve had with members of the Board, I’m putting together recommendations for how we deal with payment and PASS-branded (but not PASS run, that’s already covered in the By-Laws) events. I’ll be presenting that in June at the in-person Board meeting. Speaking of the Board meeting, I’ve also been soliciting topics (although really, my priceless partner at PASS HQ, Vicki, is doing all the real work). We’re also preparing the budget for FY 2017 (again, the real work is being led by an equally priceless Sandy at HQ). Beyond a series of meetings with each of the portfolio owners, that hasn’t been too much work for me… yet. It’s coming up and will involve quite a lot of time to have everything ready for the June meeting.

…And spreadsheets (BLECH!).

I attended the Business Analytics Conference. While I was there, instead of going to sessions (and there were a couple on R that I really wanted to attend), I sat in on Focus Group meetings with various sets of attendees in order to understand how PASS can best serve the BA community. These meetings are an excellent way to gather a ton of information from people; good feedback on how we’re doing. I’m advocating for doing them at the Summit this year too. While I was there I got to have my first in-person meeting with the rest of the Executive Committee. We talk on the phone, through Skype, and over email very regularly, but nothing beats face-to-face.

Something special I got to do, thanks again to other peoples work (in this case Annette Allen (b|t), one of the PASS Regional Mentors over in the UK) was sit down with the user group, chapter, and community leaders during SQLBits (I was over there to present a session). We talked about a whole bunch of things, but focused around how PASS, as an organization, can better serve, especially locally, like in the UK and Europe. It was a really constructive meeting and quite positive. I can share a few things from it. Based on those conversations, I’ve already set up a Slack channel that Ryan can experiment with to help improve communication between the chapters and PASS as well as the chapters with each other. I think you may also see some other changes in communication in and around chapters in the near future. We’re also going to look into ways to get more swag to Chapters (but not shipping from Canada, as international shipping is costly and efficient for PASS or organizers).

That’s about it. My involvement with the Board continues to be educational and rewarding. I won’t lie. It’s a time commitment. It is however, one I’m very happy to have the opportunity to make. If you’re considering running for the board, talk to me. I’ll tell you all the good things it brings.

May 16 2016

Query Store, Forced Plans, and New Plans

I love questions. I recently received one about new plans in the Query Store (available in Azure SQL Database now and in SQL Server 2016 after June 1).

Let’s say you have selected a plan that you want to force. You set it up. Now, let’s say the plan ages out of cache or even goes through a recompile. During the recompile, due to out of date statistics or skew in the statistics, you would, under normal circumstances, get a new plan. However, with Query Store and plan forcing, the plan that’s going to be used is the plan that is being forced. But, does that other plan, the one not used, get stored in Query Store?

I have no idea. Let’s find out.

The Setup

To start with, a small stored procedure that I use all the time for bad parameter sniffing demos that reliably gets different plans with different values due to statistics skew:

CREATE PROC dbo.spAddressByCity @City NVARCHAR(30)
AS
SELECT  a.AddressID,
        a.AddressLine1,
        a.AddressLine2,
        a.City,
        sp.Name AS StateProvinceName,
        a.PostalCode
FROM    Person.Address AS a
JOIN    Person.StateProvince AS sp
        ON a.StateProvinceID = sp.StateProvinceID
WHERE   a.City = @City;

If this procedure is called for a value of ‘London’ it gets a plan with a Merge Join. For most other value it gets a plan with a Loops Join. Here’s an example of the ‘London’ plan:

2016-02-22_10-38-43

I’ve run both queries on my test system. I can check their existence in Query Store this way:

SELECT  CAST(qsp.query_plan AS XML),
		qsq.query_id,
		qsp.plan_id,
		qsp.is_forced_plan
FROM    sys.query_store_query AS qsq
JOIN    sys.query_store_plan AS qsp
        ON qsp.query_id = qsq.query_id
WHERE   qsq.object_id = OBJECT_ID('dbo.spAddressByCity');

That’s going to return two (2) rows:

newplanresults

Now, I’m going to remove the second plan, the ‘Mentor’ plan, and I’m going to force the ‘London’ plan with two statements:

EXEC sys.sp_query_store_remove_plan @plan_id =219;

EXEC sys.sp_query_store_force_plan 2,2;

Now, when I execute the query, no matter what values I pass, I’m going to arrive at the ‘London’ plan. However, let’s pull the plan from cache in order to force a recompile:

DECLARE @PlanHandle varbinary(64);

SELECT  @PlanHandle = deps.plan_handle
FROM    sys.dm_exec_procedure_stats AS deps
WHERE   deps.object_id = OBJECT_ID('dbo.spAddressByCity');

DBCC FREEPROCCACHE(@PlanHandle);

The Reveal

When I execute the query using the value of ‘Mentor’ do I get that plan stored in the Query Store? I know which plan will be in the cache. The ‘London’ plan because of I have chosen to force that plan. But, when I query the Query Store again to see what’s there for my stored procedure:

newplannegativeresults

The short answer is, nope. Any possible new plans that could have been stored in Query Store are not there. Yes, we have gone into the compile process, but, due to plan forcing, that process is getting a small bypass and the plan is simply applied from the Query Store. This, even though, you’re going to see a compile or recompile event if you capture them using Extended Events. For more fun on recompiles and the Query Store, see this earlier post of mine.

Keep those questions coming.


I love talking about query tuning. If you want to spend the day with me talking query tuning, you sure can. In August, I’ll be doing an all day pre-con at SQLServer Geeks Annual Summit in Bangalore India.

I’ll be doing a pre-conference seminar in Oslo Norway before SQL Saturday Oslo in September.

May 12 2016

Use The Correct Data Type

Blog post #5 in support of Tim Ford’s (b|t) #iwanttohelp, #entrylevel. Read about it here.

Saying that you should use the correct data type seems like something that should be very straight forward. Unfortunately it’s very easy for things to get confusing. Let’s take a simple example from AdventureWorks. If I run this query:

SELECT  a.ModifiedDate
FROM    Person.Address AS a
WHERE   a.AddressID = 42;

The output looks like this:

2009-01-20 00:00:00.000

Normal right? You see the year, the month and the day followed by the time in hours, minutes, and seconds as a decimal. Ah, but there is an issue. This query is supposed to be for the reporting system, and the business only cares about the date that the values in the Person.Address table have been modified, so they don’t want to see the time. Also, the person in charge is kind of picky. They really don’t like seeing the date formatted that way. They prefer to see “January 20, 2009.”

Far too often then, the easy answer, just change the column to varchar. You can trim the time and output in exactly the format needed by the business. Problem solved and it was easy…

Well, until someone inputs “Janry 20, 2009” slightly mangling the spelling and suddenly your report looks all messed up. Or, they ask you to start filtering just the last two weeks, regardless of when the report was run and you now can’t easily do date math on the column. Even after you get over that problem with a little formatting using CAST (along with ISDATE to try to catch all those other typos that are in the system now)  you notice that the performance is really slow so you go to put an index on the column and now you have an index key that is 50 bytes wide instead of the 3 bytes that the DATE data type would have been, making the index less efficient (not to mention, sorting the data is going to put February ahead of January, more formatting).

The list goes on and on. You’re going to hit issue after issue and all you tried to do was a little formatting, make the data look pretty by using the VARCHAR data type in place of DATETIME or DATE. This kind of thing happens all the time, especially to people just getting started. The rules should be very simple, use the correct data type for the data you’re collecting. However, it’s easy to get distracted. It’s easy to get pushed. The business says “put it in this format” and the only way you can do that is by changing the data type so…

Don’t do it. Take the time to understand the implications of mangling the data types. It’s not just date and time that cause issues either. Lots and lots of stuff can be easily stuffed into a VARCHAR field that more properly should be an INT or a DECIMAL or some other data type. Be able to communicate with the business so that they understand that you’re not simply being difficult, you’re trying to protect the functionality and integrity of the data. Show how formatting can be done using client tools. Explain all the downstream issues that this decision will have. Then, use the correct data type for the information being stored.

May 03 2016

Implicit Conversion and Performance

Letting SQL Server change data types automatically can seriously impact performance in a negative way. Because a calculation has to be run on each column, you can’t get an index seek. Instead, you’re forced to use a scan. I can demonstrate this pretty simply. Here’s a script that sets up a test table with three columns and three indexes and tosses a couple of rows in:

CREATE TABLE dbo.ConvertTest (
     BigIntColumn BIGINT NOT NULL,
     IntColumn INT NOT NULL,
     DateColumn VARCHAR(30)
    );

CREATE INDEX BigIntIndex 
ON dbo.ConvertTest 
(BigIntColumn);
CREATE INDEX IntIndex 
ON dbo.ConvertTest 
(IntColumn);
CREATE INDEX DateIndex 
ON dbo.ConvertTest
(DateColumn);

WITH    Nums
          AS (SELECT TOP (1000000)
                        ROW_NUMBER() OVER (ORDER BY (SELECT 1
                                                    )) AS n
              FROM      master.sys.all_columns ac1
              CROSS JOIN master.sys.all_columns ac2
             )
    INSERT  INTO dbo.ConvertTest
            (BigIntColumn,
             IntColumn,
             DateColumn
            )
    SELECT  Nums.n,
            Nums.n,
            DATEADD(HOUR, Nums.n, '1/1/1900')
    FROM    Nums;

If I run a very simple query like this:

DECLARE @param DATETIME;

SET @param = '3/15/1963 8:00:00';

SELECT  ct.DateColumn
FROM    dbo.ConvertTest AS ct
WHERE   ct.DateColumn = @param;

I get an execution plan that looks like this:

convert_scan

The issue is, while I put dates into the DateColumn, the data type there is VARCHAR, not DATETIME. Passing a DATETIME value results in an implicit conversion that we can see in the Index Scan Predicate property:

ConverImplicitHurtsPerformance

This is a function against a column and that is what results in the scan. If I modify the code as follows:

DECLARE @param VARCHAR(30);

SET @param = '3/15/1963 8:00:00';

SELECT  ct.DateColumn
FROM    dbo.ConvertTest AS ct
WHERE   ct.DateColumn = @param;

I get an execution plan that looks like this:

convert_seek

No change in data type is needed, so the index can be used. It’s a very simple demonstration of how implicit conversions hurt performance. Now, what about the integer columns? What happens when I do this with the code:

DECLARE @param INT;

SET @param = 650323;

SELECT  ct.BigIntColumn
FROM    dbo.ConvertTest AS ct
WHERE   ct.BigIntColumn = @param;

Here’s the execution plan:

convert_intseek

And here is the predicate:

convert_predicate

In this case, we’re still getting an implicit conversion of the data type, but SQL Server is smart enough to realize that, hey, it’s still just an integer, so it can convert the @param instead of the column, resulting in a seek. If I change the test so that it runs a query against the INT column passing a BIGINT value, it doesn’t even do a conversion of any kind.

In short, not all implicit conversions are a problem. However, it’s still a best practice to use the right data type for columns. You should also use the same data type for your parameters and local variables as the column data type. Taking these steps avoids issues with implicit conversion.


For lots more information on query tuning, in August, I’ll be doing an all day pre-con at SQLServer Geeks Annual Summit in Bangalore India.

I’m also going to be doing an all day training course before SQL Saturday Oslo in September.

May 02 2016

How to Convince the Boss to Send You to PASS Summit

August two years ago I originally posted, Make the PASS Summit Work for Your Employer. After conversations at several SQL Saturdays over the last couple of months, I decided to refresh and update that original content and post it again.

I keep hearing how the job market has changed. That companies just don’t want to pay for training any more. However, I don’t recall any of my employers in the past ever actively wanting, desiring, begging me, please, oh, please, can’t you go out to a little training? In fact, for the most part, I pretty much always had to beg the boss to send me out to training. I had to sell it. I don’t think that’s a new development. Let’s review the selling points to help you convince the boss.

My Knowledge Base

That’s the easy one. Tell the boss, “I’ll learn more.” Maybe this one is obvious, but you should talk to your boss about the addition of more skills to your skill set, an improvement of your overall knowledge and, by extension, your worth to the company. There is a ton of excellent learning opportunities at the Summit covering the entire length, breadth and depth of Microsoft’s Data Platform and it’s attendant products. These sessions are lead by some of the most knowledgeable and skilled people in the industry. Further, they’re practically slavering at the bit to have you ask your difficult question so that they can exercise their skills and expand their knowledge by helping you. You can learn more, faster, at the PASS Summit than almost anywhere. That’s going too help your employer because you will be a better employee.

Our Current Problem

Just about every year in the 6-8 weeks leading up to the PASS Summit, I would start collecting questions. What particular pain points are we experiencing with Microsoft Data Platform products that are so severe I should grab 10 minutes with a Microsoft engineer to talk about? Oh, didn’t I mention that fact? Yeah, the guys who built the product are frequently at the Summit. You can take your immediate problems straight to these people. Further, there’s likely to be an MVP or MCM standing near by who might be able to help out too. Or, you can try the Customer Advisory Team (CAT) who always have a number of representatives there. In short, you can get pretty close to premier support without wasting a premier support ticket. All the vendors of all the tools you’re using are also there, frequently with some, or all, their development staff. Need some help with that software you purchased, go and get it.

Our Future Direction

Your company needs to make decisions about their technology future. You’ve seen the marketing hype. Now, what do the people who are working with the newest stuff every day have to say? Can you get more information by attending sessions that are not put on by Microsoft on emerging technologies? Yes, frequently. That’s not to say that a Microsoft session by the people who built the product won’t be useful too. The PASS Summit is the place to see this. Microsoft doesn’t just develop things and then toss them over the fence to see what works (mostly). Instead, they have companies and individuals working with them all the time to develop new directions for the product. Those people and organizations are frequently at the Summit, displaying new stuff on the vendor floor or giving presentations about the new directions they’re taking the technology. You can get a better understanding if your company’s plans are going to work well going into the future. Even if the plan is best summed up as “We’ll sit on SQL Server 2000 until it rots around our ears.” Others are doing it too. Find out how it’s working out for them. Or, why they finally decided to upgrade, maybe even moving to Azure.

Our Team Skill Set

Most companies are not going to want to send all of the database development team, DBA team, or development team away for a week. Instead, they’ll send one or two people from each team (maybe less). So your team loses out, right? Wrong. Two things. First, coordinate. If you have more than one person from your company at the event, make sure that you cover as many sessions as you possible can. Don’t overlap. When I was working on a team heading to the Summit we would divide up sessions to make sure things got covered that the company needed or that we needed as individuals. While I may want to see speaker X do her session on indexing again, my co-worker has yet to see it, so I’ll send them. And make sure you have a couple of sessions picked for a time period because the session you’re in could be a bad choice. If a session isn’t for you, for any reason, just walk out. Before you go, if you’re the only one going, head around to the teams and see if they have a request for a session that you can attend. This is a chance to enhance your image within the organization and make your boss look good by offering to help others. Send them links to the event schedule so that they can pick and choose. Finally, teach. You just spent a week getting data dumped into your brain. Teach it to your team. We made a pact that anyone who went off to a week of training had to present 2-3 sessions to the team from that event. You can even purchase the event DVD and show sessions to your team in meetings.

NOTE: This is not to say, steal these slide decks to become your internal training resource, unattributed to the original presenter. That is a bad thing.

My Retention

Who do you want to work for? The employer that says, “Heck no you can’t go to the PASS Summit. You’ll meet people and figure out that our company stinks and you’ll try to get a new job, or you’ll learn more and be more valuable and we’re not about to raise your pay.” Or, the employer who says, “Yeah, sure you can go this year. Let’s document what you’re going to learn and how it’ll help the company.” OK, it’s not going to be that easy. You may have to agree not to leave the company for a year or something afterwards. Be cautious about exactly what kind of strings get attached, but also be aware of the fact that the company is investing in you and would probably expect to get something for that investment. Just be sure it’s fair to both you and them.

I get it that some employers are smaller and just can’t foot the bill for this. See if they’ll meet you part way. You pay for the trip and lodging and they pay for the Summit, or vice versa. It can also be about timing. You’ve got a major software release that’s going to prevent you from going. I almost missed a Summit myself because of this. It’s just not always possible, but a good employer will find a way to make it possible, occasionally. If there is literally no support, of any kind, ever, you’re either working for a not-for-profit or, maybe, the wrong company.

I’ll Be On Call

Be on call. Carry the laptop with you. Keep your phone charged (ABC = Always Be Charging). Don’t enjoy the evening festivities too much (and yes, there are parties at the PASS Summit). Be a responsible employee. I’ve had to walk out of great sessions because of calls from the office. I missed half a day because of a failed deployment. But I was online and available, not falling off the face of the planet just because I was at the Summit. Make the commitment to be available as needed by your employer. Demonstrate that commitment by being available. However, as with all things, there has to be a happy middle, assuming a non-destructive total emergency, they should leave you alone for little stuff so that you can attend sessions and network. That’s why they sent you in the first place.

My Notes

Take lots and lots and lots of notes. You can type them into OneNote or EverNote or whatever. Or you can scribble them into your tablet or onto notepads. Anything that works. But write stuff down. Write lots of stuff down. Write down what you’re thinking about the information as well as details said by the speaker that may not be visible on slides or in code. Write down what you talked about with that lady from that vendor on the back of their card. Take notes while talking to the Microsoft engineer or CAT member. Then, turn the notes over to your employer. They act as an additional knowledge base about the event. It’s one more resource that you’re bringing back to your team, showing the enhanced value that you’re providing.

Our Swag

Bring home a t-shirt or two for those people who couldn’t go. If there’s a particularly cool piece of swag, give it to the boss or have it as a raffle at the team training event for the best question. Share the stuff you get as well as the information you get. A friend of mine and I once collected 56 t-shirts and a stack of other swag (and had a heck of a time getting it all back on the plane) which we then spent almost two weeks handing out in the office to our team, development teams, managers and systems people, etc. It made us look good and cost us nothing but a little time on the vendor floor. It’s silly, but it works. If nothing else, it shows the boss that you’re thinking about your team and the company while you’re away.

My/Our Network

Network. That means not being “that person.” That person is the one who comes to the event, shows up for all the sessions, doesn’t ask questions or talk to a single person all day, then leaves and goes to their hotel room (and then usually goes home saying “Wow, that was a waste of my time”). There are large numbers of opportunities to network. Waiting in line to register, turn and talk to someone. Ask questions of the presenter during their session AND follow-up afterwards (although, let them get unplugged and out of the way of the next speaker). Go to the vendor floor where you should talk to the vendors as well as others. Attend the First-Timers event. Go to the Birds of a Feather lunch. Wear a kilt on Day 2 of the Summit (SQL Kilt Day, you’re reading the words of the founder of the event). Attend the Women in Technology Luncheon. Track down all the places where people are getting together and talking. Go to them. Get together. Talk.

I’m an introvert (people laugh when I say it, but it’s true). I recharge with alone time, not at parties. I get being an introvert. But the PASS Summit is not recharge time. If you’re not almost literally crawling out of the venue on Friday afternoon, you’re doing it wrong. The flight home should be the most relaxing plane flight you’ve ever had because you’ll pass out before take-off and wake up when the wheels touch down.

Take the time and trouble to begin to build your network. And remember, a network is not a series of authors or MCMs or MVPs that you can call. It’s a collection of people, some may be presenters/authors/etc., but the best are probably doing the same job you do but for a different organization. Talk to everyone. Build that network.

How does your network help the company? Remember that you don’t know everything. You can’t. However, you can know the people who do know things that you do not. That effectively expands your knowledge set. That makes you more valuable for your organization.

Conclusion

As you can see, going to the event could be a ton of work. In fact, if you’re focused on maximizing the returns for your organization, it will be. You’re going to be working just as hard at this event as you do in the office. It’s all about showing the organization that they will receive benefits by sending you. They will profit from the expenditure. Never lose sight of the fact that it has to be a partnership with the business. You need to benefit as much as they do from the experience. The fact is though, if you follow all my suggestions, you will benefit, and you will deliver worth to your org.

Apr 27 2016

Customizing Your Azure Portal

Not all of us are going to use Azure in the same way. That’s just common sense. Therefore, not all of us are going to want it to look the same. Here’s a default view of the portal for a login I created within my own domain:

AzurePortalDefault

You can contrast that with this login to my portal that I use for most of my testing and training (this isn’t my company’s production or development portal, which looks still more different):

AzurePortalCustomized

Clicking on either of these will open them so you can look at details. Not only are the color schemes different, but you’ll note that the selection lists on the left are different as are the grids on the dashboard. All this can be customized for each login, and, within a login, on each Subscription to which that login belongs.

All this means that you can customize the portal for your individual needs, even between your production and development areas within the portal (assuming you’re keeping them separate, which you ought to be). Let’s explore a few of those options.

Favorites

Over on the left of the screen, is a listing of favorite resources. At the bottom of this list you’ll see a “Browse >” button. Selecting this pops out a list of all available resources:

AzureportalFavorties

This is a complete listing of the resources available to your subscription, including anything that is in Preview. Note, on the right hand side of the screen are stars. One of the stars in the list above shows up as gold and filled in. This indicates that it’s currently on my list of resources as a favorite. Selecting one of the resources that is not filled in will add it to the list and selecting one that is filled in will remove it. This is your first customization. Add the resources to favorites that you use all the time. You can even drag them around to rearrange the order in which they appear.

Portal Settings

If you want to change the color scheme, you can do that too. At the top of the screen is a little icon of a gear. Clicking on that opens the Portal Settings blade:

azureportalsettings

Not only can you modify the colors scheme, but you get to set labels, animations & the language of the portal.

Dashboard

This is where you can get crazy. You have the ability to add all sorts of output, readouts, shortcuts, to your dashboard. You get to set their size, rearrange them, even create additional dashboards. Additional dashboards give you the ability to customize a view so that you could, for example, have your most important Azure SQL Databases displaying their DTU readouts as a single view:

azureportalNewDashboard

Yeah, yeah, it reads zero. Test databases, remember. Anyway, customizing a dashboard is easy. If you click on “+New dashboard” or “edit dashboard” at the top of the screen, a new blade opens. This allows you to add and rearrange tiles. You’ll see a grid that has dashed lines for the resolution of a screen. It makes it very easy to figure out what’s going to be displayed and what’s not. On the left you get the ability to track down different kinds of tiles. In the example below I have a list of the tiles for Azure SQL Database (you will need to click on this one, it’s big):

azureportaldashboardedit

You can drag and drop the tiles from the left on to the screen. Once there you can rearrange where they are located, resize them, remove them, pretty much what you would expect. Further, you can customize each tile individually. The tile will have a clickable link called “Edit”. Clicking on that will bring up unique options depending on the tile. In the example below I’m showing the options for the Resource Utilization tile:

azureportaleditchart

I’ve decided to just show DTU percentage and DTU used for the last hour as a Line chart. You can rearrange this as necessary to fit your own needs. You can rename the dashboard, and then, when you’re done, just click on the “Done Customizing” button and you’ll see the dashboard. You can pick which dashboard to look at. You can even share a dashboard with your team so that everyone has access to the same information you do.

Conclusion

This was just a bare bones run-through of what’s possible with the Azure portal. You really can achieve a very high level of customization to get just the information you need, in the way that you need it. Take advantage of this because, at times, it can be quite difficult to manage information on the portal.

Apr 21 2016

Microsoft’s Commitment to Azure

For several years, many of us who were involved with working in Azure from the early days, were concerned that Microsoft had bet the farm and could possibly lose it all. They may well have bet the farm, but more and more it’s extremely clear that there is zero likelihood of them losing the farm. In fact, it’s looking more and more like, while using farming analogies, they’re turning into an agro-corp. Azure is taking off.

You need to start working on adding Azure knowledge to your skill set. If you have access to an MSDN license, getting into Azure is easy because of the credits available. However, not everyone works for a company that provides MSDN or has purchased a license. In that case, getting into Azure, just for testing and learning could be expensive (I frequently “spend” $150/month with my MSDN credits). However, Microsoft is very serious about getting everyone moved into this space. They’ve launched a new free program called IT Pro Cloud Essentials. Not only does it give you Azure credit, but you also get access to O365, another set of skills and understanding you need to acquire.

If cost was one of the reasons you were choosing to not explore Azure, time to give that one up. Explore the offering and then get your learn on.