11

ISQL Users Group May 2009ISQL Users Group May 2009

Tracing and Profiling Tracing and Profiling SQL ServerSQL Server

Yaniv.Etrogi@gmail.comYaniv.Etrogi@gmail.comhttp://blogs.microsoft.co.il/blogs/yaniv_etrogihttp://blogs.microsoft.co.il/blogs/yaniv_etrogi

22

Over the next 1 hour:Over the next 1 hour:

SQL Trace Architecture and SQL Trace Architecture and TerminologyTerminology

Security and Permissions Security and Permissions

ProfilerProfiler

Server side TracingServer side Tracing

Saving and Replaying TracesSaving and Replaying Traces

Querying Server-Side Trace MetadataQuerying Server-Side Trace Metadata

33

IntroductionIntroduction

Query tuning, optimization and Query tuning, optimization and troubleshooting are all possible by the troubleshooting are all possible by the ability to view what's going on inside ability to view what's going on inside SQL ServerSQL Server

SQL Trace provides a real-time view SQL Trace provides a real-time view into what’s going on inside the database into what’s going on inside the database engine and at a very granular level engine and at a very granular level

44

SQL Trace Architecture and SQL Trace Architecture and Terminology Terminology

55

SQL Trace is an SQL Server SQL Trace is an SQL Server database engine technologydatabase engine technology

SQL Server Profiler is a .NET SQL Server Profiler is a .NET application that uses system application that uses system stored procedures exposing the stored procedures exposing the functionality of SQL Tracefunctionality of SQL TraceMicrosoft.SqlServer.Management.TraceMicrosoft.SqlServer.Management.Tracehttp://msdn.microsoft.com/en-us/library/http://msdn.microsoft.com/en-us/library/microsoft.sqlserver.management.trace.aspxmicrosoft.sqlserver.management.trace.aspx

66

When tracing we monitor for specific When tracing we monitor for specific events which are generated when various events which are generated when various actions occur in the database engineactions occur in the database engine

– For example, a user login or the execution of a query are For example, a user login or the execution of a query are each actions that cause events to fireeach actions that cause events to fire

Each event has an associated collection of Each event has an associated collection of columns that contain the collected data columns that contain the collected data when the event fireswhen the event fires

SQL Server uses a model that selectively SQL Server uses a model that selectively enables data collection as requiredenables data collection as required

– Data is never collected until someone asks for itData is never collected until someone asks for it

There are around 171 events and 65 There are around 171 events and 65 columns to select fromcolumns to select from

88

Trace I/O providers send the data to the Trace I/O providers send the data to the final destinationfinal destination

The available output formats for trace The available output formats for trace data are either a file on the database data are either a file on the database server (or a network share) or a rowset server (or a network share) or a rowset to a clientto a client

Both providers use internal buffers to Both providers use internal buffers to ensure that if the data is not consumed ensure that if the data is not consumed quickly enough (written to disk or read quickly enough (written to disk or read from the rowset) it will be queuedfrom the rowset) it will be queued

99

The file provider is designed with a The file provider is designed with a guarantee that no event data will be lostguarantee that no event data will be lost

If an I/O slowdown occurs so disk writes If an I/O slowdown occurs so disk writes are not occurring quickly enough the are not occurring quickly enough the internal buffers begin to fillinternal buffers begin to fill

To monitor for these waits watch the To monitor for these waits watch the SQLTRACE_LOCK and IO_COMPLETION SQLTRACE_LOCK and IO_COMPLETION wait typeswait types

1010

The rowset provider is not designed for data The rowset provider is not designed for data loss guaranteeloss guarantee

If data is not being consumed quickly enough If data is not being consumed quickly enough and its internal buffers fill it waits up to 20 and its internal buffers fill it waits up to 20 seconds before it begins dropping events in seconds before it begins dropping events in order to free buffers for the sake of getting order to free buffers for the sake of getting things movingthings moving

SQL Server Profiler will then send an SQL Server Profiler will then send an information message informing that some information message informing that some events had been lost and had not been events had been lost and had not been capturedcaptured

You can find out if you're headed in that You can find out if you're headed in that direction by monitoring SQL Server's direction by monitoring SQL Server's TRACEWRITE wait type which is incremented as TRACEWRITE wait type which is incremented as threads are waiting for buffers to free upthreads are waiting for buffers to free up

1111

A background trace management A background trace management thread is started whenever at least one thread is started whenever at least one trace is active on the servertrace is active on the server

– SELECT * FROM sysprocesses SELECT * FROM sysprocesses WHERE cmd = N'TRACE QUEUE TASK' ;WHERE cmd = N'TRACE QUEUE TASK' ;

The background thread is responsible The background thread is responsible for:for:

– Flushing file provider buffers (done every 4 seconds) Flushing file provider buffers (done every 4 seconds) – Can be seen as a wait_type of Can be seen as a wait_type of SQLTRACE_BUFFER_FLUSHSQLTRACE_BUFFER_FLUSH– Closing rowset-based traces that are considered to be Closing rowset-based traces that are considered to be expired (dropping events for more than 10 minutes)expired (dropping events for more than 10 minutes)

There is no provider that writes trace There is no provider that writes trace data directly to a tabledata directly to a table

1212

Security and PermissionsSecurity and Permissions

1313

SQL Server 2005 introduces a new SQL Server 2005 introduces a new permission called ALTER TRACE. This is a permission called ALTER TRACE. This is a server-level permission granted to a login server-level permission granted to a login principal and allows access to start, stop, or principal and allows access to start, stop, or modify a trace (in addition to being able to modify a trace (in addition to being able to generate user-defined events)generate user-defined events)

GRANT ALTER TRACE TO GRANT ALTER TRACE TO <LoginName><LoginName>;;

SQL Trace has a couple of built-in security SQL Trace has a couple of built-in security features to keep passwords securedfeatures to keep passwords secured

–SQL Trace will automatically omit data if an event SQL Trace will automatically omit data if an event contains a call to a password-related stored procedure or contains a call to a password-related stored procedure or statement. For example, a call to CREATE LOGIN including statement. For example, a call to CREATE LOGIN including the WITH PASSWORD option will be blanked out by SQL the WITH PASSWORD option will be blanked out by SQL Trace. Trace. –SQL Trace will not return statement text or query plans SQL Trace will not return statement text or query plans generated within an encrypted stored procedure, user-generated within an encrypted stored procedure, user-defined function, or viewdefined function, or view

1414

ProfilerProfiler

1515

The General tab allows you to control how the trace will be The General tab allows you to control how the trace will be processed by the consumerprocessed by the consumer

The default setting is to use the rowset provider displaying The default setting is to use the rowset provider displaying the events in real time in the SQL Server Profiler tool windowthe events in real time in the SQL Server Profiler tool window

Other available options areOther available options are –Save the events to a tableSave the events to a table–Save the events to a file (on either the server or the client)Save the events to a file (on either the server or the client)

When saving to a table Profiler uses the rowset provider and When saving to a table Profiler uses the rowset provider and routes the data back into a tableroutes the data back into a table

When saving to a server side file Profiler actually starts two When saving to a server side file Profiler actually starts two equivalent traces—one using the rowset provider and the equivalent traces—one using the rowset provider and the other using the file providerother using the file provider

Saving to a client-side file does not use the file provider at Saving to a client-side file does not use the file provider at all. Rather, the data is routed to the Profiler tool via the all. Rather, the data is routed to the Profiler tool via the rowset provider and then saved from there to a file.rowset provider and then saved from there to a file.

–This is more efficient than using Profiler to write to a server-side fileThis is more efficient than using Profiler to write to a server-side file–Remember that doing so you do not take benefit of the lossless data guarantee Remember that doing so you do not take benefit of the lossless data guarantee offered by the file provideroffered by the file provider

1616

The Events Selection tab allows you to select events that The Events Selection tab allows you to select events that you'd like to trace along with their associated data columns you'd like to trace along with their associated data columns

In order to narrow your scope and help ensure that tracing In order to narrow your scope and help ensure that tracing does not cause performance issues SQL Trace offers the does not cause performance issues SQL Trace offers the ability to filter the events based on various criteria ability to filter the events based on various criteria

In SQL Server Profiler filtration is exposed via the “Column In SQL Server Profiler filtration is exposed via the “Column Filters” tabFilters” tab

Remember to check the “Show All Columns” checkbox in Remember to check the “Show All Columns” checkbox in order to see the complete list of columnsorder to see the complete list of columns

Once you click the Run button data will begin streaming Once you click the Run button data will begin streaming immediately and be displayed at the Profiler window (this is immediately and be displayed at the Profiler window (this is because Profiler uses the rowset provider)because Profiler uses the rowset provider)

–If you find that data is coming in too quickly for you to be able to read it If you find that data is coming in too quickly for you to be able to read it you may disable scrolling using the Auto Scroll icon on the SQL Server you may disable scrolling using the Auto Scroll icon on the SQL Server Profiler toolbarProfiler toolbar–If you are on a slow remote connection and your trace is not well filtered If you are on a slow remote connection and your trace is not well filtered you may be in a situation that the icons in the Profiler tool bar are seen you may be in a situation that the icons in the Profiler tool bar are seen as if they were inactive thus disallowing you to stop the trace. The only as if they were inactive thus disallowing you to stop the trace. The only way out here is to stop the trace using way out here is to stop the trace using sp_trace_setstatussp_trace_setstatus

1818

Saving and Replaying Saving and Replaying TracesTraces

1919

Profiler ships with eight predefined templates Profiler ships with eight predefined templates –A template is a collection of event and column selections, filters, and other A template is a collection of event and column selections, filters, and other settings that you can save to create a reusable trace definitionssettings that you can save to create a reusable trace definitions

TSQL_Replay template selects a variety of columns for 15 events TSQL_Replay template selects a variety of columns for 15 events that are all required for Profiler to be able to play back (Replay) a that are all required for Profiler to be able to play back (Replay) a collected trace at a later time. collected trace at a later time.

–Collecting data using this template allows you to reproduce a problem Collecting data using this template allows you to reproduce a problem experienced on a production system by replaying events collected on the experienced on a production system by replaying events collected on the production system at a test environment production system at a test environment –Very useful for support departmentsVery useful for support departments

Collected trace data has to be saved and then reopened before a Collected trace data has to be saved and then reopened before a replay can beginreplay can begin

SQL Server Profiler offers the following options for saving trace SQL Server Profiler offers the following options for saving trace data available from the File menu:data available from the File menu:

–The Trace File option is used to save the data to a file formatted using a The Trace File option is used to save the data to a file formatted using a proprietary binary format. This is generally the fastest way to save the data, and proprietary binary format. This is generally the fastest way to save the data, and also the smallest in terms of bytes on diskalso the smallest in terms of bytes on disk–The Trace Table option is used to save the data to a new or previously created The Trace Table option is used to save the data to a new or previously created table in a database. This option is useful if you need to manipulate or report on the table in a database. This option is useful if you need to manipulate or report on the data using T-SQLdata using T-SQL–The Trace XML File option saves the data to a text file formatted as XMLThe Trace XML File option saves the data to a text file formatted as XML–The Trace XML File For Replay option also saves the data to an XML text file, but The Trace XML File For Replay option also saves the data to an XML text file, but only those events and columns needed for replay functionality are savedonly those events and columns needed for replay functionality are saved

2020

Once the data has been saved to a file or table the original Once the data has been saved to a file or table the original trace window can be closed and the file or table reopened via trace window can be closed and the file or table reopened via SQL Server Profiler's File menuSQL Server Profiler's File menu

A trace reopened in this way will have a Replay menu on A trace reopened in this way will have a Replay menu on the Profiler tool bar allowing you to start replaying the trace, the Profiler tool bar allowing you to start replaying the trace, stop the replay, or set a breakpointstop the replay, or set a breakpoint

During the course of the replay, the same events used to During the course of the replay, the same events used to produce the trace being replayed will be traced from the produce the trace being replayed will be traced from the server on which you replayserver on which you replay

–The ”Save To File” and “Save To Table” options are used for a client-side The ”Save To File” and “Save To Table” options are used for a client-side save. No server-side option exists for saving playback resultssave. No server-side option exists for saving playback results

The trace will be replayed on multiple threads (2005 only), The trace will be replayed on multiple threads (2005 only), corresponding to at most the “Number Of Replay Threads” corresponding to at most the “Number Of Replay Threads” specifiedspecified

–The “Replay Events In The Order They Were Traced” option ensures that The “Replay Events In The Order They Were Traced” option ensures that all events will be played back in exactly the order in which they occurred all events will be played back in exactly the order in which they occurred as based upon the EventSequence column. Multiple threads will still be as based upon the EventSequence column. Multiple threads will still be used to simulate multiple spids. used to simulate multiple spids. –The “Replay Events Using Multiple Threads” option allows SQL Server The “Replay Events Using Multiple Threads” option allows SQL Server Profiler to reorder the order in which each spid starts to execute events, Profiler to reorder the order in which each spid starts to execute events, in order to enhance playback performance. However, the order of events in order to enhance playback performance. However, the order of events will remain consistent with the EventSequence within a given spidwill remain consistent with the EventSequence within a given spid

2121

Server-Side TracingServer-Side Tracing

2222

sp_trace_createsp_trace_create is used to define a trace and is used to define a trace and specify an output file location. This stored specify an output file location. This stored procedure returns a handle to the created trace procedure returns a handle to the created trace in the form of an integer TraceIdin the form of an integer TraceId

sp_trace_seteventsp_trace_setevent is used to add/remove is used to add/remove event/column combinations to traces based on event/column combinations to traces based on the TraceIdthe TraceId

sp_trace_setfiltersp_trace_setfilter is used to define event is used to define event filters based on trace columnsfilters based on trace columns

sp_trace_setstatussp_trace_setstatus is called to start, stop and is called to start, stop and and delete a trace. and delete a trace.

–Traces can be started and stopped multiple times over Traces can be started and stopped multiple times over their lifespantheir lifespan

2323

sp_trace_setfiltersp_trace_setfilter @TraceID@TraceID,@columnid,@columnid,@logical_operator,@logical_operator,@comparison_operator,@comparison_operator,@value ,@value

/*/*@logical_operator: AND (0) or OR (1) @logical_operator: AND (0) or OR (1)

@comparison_operator:@comparison_operator:0 = Equal, 1 = Not equal, 2 = Greater than, 3 = 0 = Equal, 1 = Not equal, 2 = Greater than, 3 = Less than, 4 = Greater than or equal, Less than, 4 = Greater than or equal, 5 = Less than or equal, , 6 = Like, 7 = Not like 5 = Less than or equal, , 6 = Like, 7 = Not like */*/

2525

Querying Server-Side Trace Querying Server-Side Trace MetadataMetadata

2626

Get TracesGet Traces

SELECT SELECT **

FROM sys.traces WHERE [status] = 1FROM sys.traces WHERE [status] = 1;; --This query returns the trace status, which will --This query returns the trace status, which will

be 1 (started) or 0 (stopped); the server-side be 1 (started) or 0 (stopped); the server-side path to the trace file (or NULL if the trace is path to the trace file (or NULL if the trace is using the rowset provider); the maximum file using the rowset provider); the maximum file size (or again, NULL in the case of the rowset size (or again, NULL in the case of the rowset provider); information about how many buffers provider); information about how many buffers of what size are in use for processing the I/O; of what size are in use for processing the I/O; the number of events captured; and the the number of events captured; and the number of dropped events (NULL if your trace number of dropped events (NULL if your trace is using the file provider(loseless guarantee) ).is using the file provider(loseless guarantee) ).

2727

Get EventsGet Events

SELECTSELECTe.name AS e.name AS [e[eventvent]],,c.name AS c.name AS [c[columnolumn]]

FROM fn_trace_geteventinfo(FROM fn_trace_geteventinfo(@TraceId@TraceId) ei) eiINNER INNER JOIN sys.trace_events e JOIN sys.trace_events e

ON ei.eventid = e.trace_event_idON ei.eventid = e.trace_event_idINNER INNER JOIN sys.trace_columns c JOIN sys.trace_columns c

ON ei.columnid = c.trace_column_id;ON ei.columnid = c.trace_column_id;

2828

Get FiltersGet Filters

SELECTSELECT columnidcolumnid,c.name AS ,c.name AS [c[columnolumn]],logical_operator,logical_operator,comparison_operator,comparison_operator,value,value

FROM fn_trace_getfilterinfo(FROM fn_trace_getfilterinfo(TraceId TraceId ) ei) eiINNER INNER JOIN sys.trace_columns c JOIN sys.trace_columns c

ON ei.columnid = c.trace_column_id;ON ei.columnid = c.trace_column_id;

2929

Get Events/Columns combinationGet Events/Columns combination

SELECT SELECT

e.name AS [event]e.name AS [event]

,c.name AS [column],c.name AS [column]

FROM sys.trace_event_bindings bFROM sys.trace_event_bindings b

INNER JOIN sys.trace_events e INNER JOIN sys.trace_events e

ON e.trace_event_id = b.trace_event_id ON e.trace_event_id = b.trace_event_id

INNER JOIN sys.trace_columns c INNER JOIN sys.trace_columns c

ON c.trace_column_id = b.trace_column_id ON c.trace_column_id = b.trace_column_id

ORDER BY e.nameORDER BY e.name;;

3030

Get Get SubClass valuesSubClass values

SELECT SELECT c.name AS [column]c.name AS [column],e.name AS [event],e.name AS [event],s.subclass_value,s.subclass_value,s.subclass_name,s.subclass_name

FROM sys.trace_columns cFROM sys.trace_columns cINNER JOIN sys.trace_subclass_values s INNER JOIN sys.trace_subclass_values s ON c.trace_column_id = s.trace_column_id ON c.trace_column_id = s.trace_column_id INNER JOIN sys.trace_events e INNER JOIN sys.trace_events e ON e.trace_event_id = s.trace_event_id ON e.trace_event_id = s.trace_event_id WHERE e.name LIKE 'Audit Login';WHERE e.name LIKE 'Audit Login';`̀

3131

Reading trace filesReading trace files

SELECT * FROM fn_trace_gettable(‘B:\Traces\TraceErrors.trc', 1);SELECT * FROM fn_trace_gettable(‘B:\Traces\TraceErrors.trc', 1);–This function takes two parameters: The path + name to the trace file and This function takes two parameters: The path + name to the trace file and the number of trace files to read. If the second parameter is DEFAULT then all the number of trace files to read. If the second parameter is DEFAULT then all available trace files will be readavailable trace files will be read

SELECT * INTO ProfilerErrors SELECT * INTO ProfilerErrors FROM fn_trace_gettable(‘B:\Traces\TraceErrors.trc', FROM fn_trace_gettable(‘B:\Traces\TraceErrors.trc', DEFAULT);DEFAULT);

–Loading the file to a table allows querying the data.Loading the file to a table allows querying the data.–It also consumes less resources as it is a one time operation and benefits It also consumes less resources as it is a one time operation and benefits from the fact that the SELCT..INTO command is a BULK operation where as from the fact that the SELCT..INTO command is a BULK operation where as just performing queries against the trace loads sql server’s memory and if it just performing queries against the trace loads sql server’s memory and if it is a large set of data spilling begins thus incurring additional I/Ois a large set of data spilling begins thus incurring additional I/O

SELECT IDENTITY(int, 1, 1) AS RowNumber, * SELECT IDENTITY(int, 1, 1) AS RowNumber, * INTO dbo.ProfilerErrors INTO dbo.ProfilerErrors FROM fn_trace_gettable(‘B:\Traces\TraceErrors.trc', FROM fn_trace_gettable(‘B:\Traces\TraceErrors.trc', DEFAULT);DEFAULT);

3232

Controlling trace stateControlling trace state

EXECEXEC sp_trace_setstatussp_trace_setstatus @TraceId, 1;@TraceId, 1; --start --startEXECEXEC sp_trace_setstatussp_trace_setstatus @TraceId, 0;@TraceId, 0; --stop --stopEXECEXEC sp_trace_setstatussp_trace_setstatus @TraceId, 2;@TraceId, 2; –remove –remove

A trace in a stopped state can be started without the A trace in a stopped state can be started without the need to recreate the trace. This allows to modify the need to recreate the trace. This allows to modify the Events/Columns selection and the selected FiltersEvents/Columns selection and the selected Filters

A restart of the SQL Server service removes all trace A restart of the SQL Server service removes all trace definitionsdefinitions

–Zeros the Zeros the EventSequenceEventSequence column column–If you need your trace always up and running you can If you need your trace always up and running you can launch it from a start up stored procedurelaunch it from a start up stored procedure

3333

Import Performance Monitor DataImport Performance Monitor Data

Default TraceDefault Trace

Balackbox TraceBalackbox Trace

3434

??

3535

SQL Server BOL (Books Online)SQL Server BOL (Books Online)

sys.dm_os_wait_stats sys.dm_os_wait_stats http://msdn.microsoft.com/en-us/library/ms179984.aspxhttp://msdn.microsoft.com/en-us/library/ms179984.aspx

Troubleshooting and Analysis with Traces Troubleshooting and Analysis with Traces http://technet.microsoft.com/en-us/library/cc293616.aspxhttp://technet.microsoft.com/en-us/library/cc293616.aspx

Introducing SQL Trace Introducing SQL Trace http://msdn.microsoft.com/en-us/library/ms191006.aspxhttp://msdn.microsoft.com/en-us/library/ms191006.aspx

SQL Profiler EventsSQL Profiler Eventshttp://msdn.microsoft.com/en-us/library/ms186265.aspxhttp://msdn.microsoft.com/en-us/library/ms186265.aspx

SQL Profiler Data Columns SQL Profiler Data Columns http://msdn.microsoft.com/en-us/library/aa173882(sql.80).aspxhttp://msdn.microsoft.com/en-us/library/aa173882(sql.80).aspx

Inside Microsoft SQL Server 2005 Inside Microsoft SQL Server 2005 http://www.insidesqlserver.com/thebooks.htmlhttp://www.insidesqlserver.com/thebooks.html

3636

Trace Errors Trace Errors

http://blogs.microsoft.co.il/blogs/yaniv_etrogi/TraceErrors.ziphttp://blogs.microsoft.co.il/blogs/yaniv_etrogi/TraceErrors.zip