Blocking Lock Killer - CoHEsion Nashville 2013
description
Transcript of Blocking Lock Killer - CoHEsion Nashville 2013
Presented by: Shawn Lee, University of Illinois
Monday, November 4, 2013Session #149
Blocking Lock Killer
co•he•sion noun \kō-ˈhē-zhən\ 1 : the act or state of sticking together tightly; especially: unity
2
TopicsDatabase transactions and locking mechanisms
Blocking locks in Banner
Monitoring & triaging blocking locks
University of Illinois’ blocking lock killer tool
CoHEsion Summit
CoHEsion Summit3
Audience Participation Test!
Overview of database transactions and locking mechanisms
Concepts
5
What is a transaction?A single execution within the database
A series of reads and writes of database objects
Final action – commit or abort
Committed transactions must be durable
Aborted transactions must be undone
CoHEsion Summit
6
ACIDTransactions have four essential properties:
1) atomicity
2) consistency
3) isolation
4) durability
CoHEsion Summit
7
Motivation for ConcurrencyTransactions can be executed in serial, but…that’s ineffiecient
DBMS improves performance by interleaving the actions of different transactions so they can run concurrently
Only serializable schedules are generated
CoHEsion Summit
8
Issues with ConcurrencyThree problems that can arise:
Dirty Reads (WR conflict) – a transaction reads a value written by another uncommitted transaction
Unrepeatable Reads (RW conflict) – a transaction reads the value of the same object twice; another transaction modifies that value in between the two reads
CoHEsion Summit
9
Issues with Concurrency (cont.)Lost Updates (WW conflict) – Two transactions update the value of the same object, before either of them have committed
CoHEsion Summit
10
Data Locking MechanismsShared Locks – acquired when reading data without making modifications; shared with other transactions that only need to read, not write
Exclusive Locks – acquired when modifying data, prevents any other transaction from accessing the data
The amount of data locked can vary
CoHEsion Summit
11
More on lockingTwo-Phase Locking (2PL) – common protocol used to make it easier for DBMS to schedule serializable transactions
# of locks acquired by transaction grows (phase 1) then shrinks (phase 2)
Deadlocks – when two transactions are stuck waiting on each other; Oracle detects and handles these automatically
CoHEsion Summit
Database Locks in BannerCauses of blocking locks and their
effects
13
The effect of locks in BannerDatabase locks are commonLocks can last for seconds, minutes, or hours depending on what the root blocker is doingThe “blocker” and “blockee” could be a batch job, or a user.If the blockee is a user, an extensive lock can result in a poor experience
CoHEsion Summit
14
Common CausesUsers blocking themselves
SOAHOLD (placing student holds)PEALEAV (employee leave balances/accruals)FZAREQN (custom form for multiyear POs)
Peak processing times (start of term activities, grade roll, payroll deduction runs)
Scheduled jobs (Appworx, a.k.a. UC4)
CoHEsion Summit
CoHEsion Summit15
Poll
Triaging Blocking LocksHow we handle them
17
It starts with active monitoringHP SiteScopeselect count(*) from v$session a, v$session b where 10 <= round(a.seconds_in_wait/60,1) and a.blocking_session_status = 'VALID’ and a.wait_class = 'Application’ and a.blocking_session=b.sid and b.username not like 'APPWORX%'
CoHEsion Summit
18
Our triage processMagic number = 10 minutesIdentify the root blockerDuring business hours, call user firstIf can’t make phone contact, kill sessionExtra care for jobsub processesSelf-service sessions killed immediatelyOutside business hours, kill immediately then communicate immediately
CoHEsion Summit
19
After the killThe database rolls back the transaction
User may lose work
Forms users screen may go blank and/or they will get network errors
Impact to self-service users will be less due to web session persistence
Communication and follow-up are keyCoHEsion Summit
The Tool The old and the new
21 CoHEsion Summit Session ID# 149
The Old Tool
22 CoHEsion Summit Session ID# 149
The Old Tool (cont.)
23 CoHEsion Summit Session ID# 149
The New Tool
24 CoHEsion Summit Session ID# 149
Session Table
25 CoHEsion Summit Session ID# 149
Contact Info
26 CoHEsion Summit Session ID# 149
Details and Kill Now options
27 CoHEsion Summit Session ID# 149
Session Details
28 CoHEsion Summit Session ID# 149
After the Kill
29 CoHEsion Summit Session ID# 149
Tool Permissions
WebTailor user roles
System Admin – Can Kill Sessions
System Monitor – “Kill” option not visible
30
Database procedures-- FILE NAME..: pkg_banner_monitoring_tools.pkg-- OBJECT NAME: UILLINOIS.pkg_banner_monitoring_tools-- USAGE......: Banner monitoring tools, including new blocking lock utility to replace the blocking lock utility from the-- Triage site that was written by Mike Cornell
procedure P_ShowSessionDetails (sid_in IN v$session.sid%TYPE, serial_in IN v$session.serial#%TYPE);
procedure P_ShowSessionsAndBlocks (dispinfo IN VARCHAR2 DEFAULT 'DEFAULT');
procedure P_KillSession (sid_in IN v$session.sid%TYPE, serial_in IN v$session.serial#%TYPE, username_in IN v$session.username%TYPE);
CoHEsion Summit
31
Noteworthy components
CoHEsion Summit
Database cursor – blocker and blockee information in each rowV$LOCK, V$SESSION, V$PROCESSContact info retrieved from GOBEACC, SPRIDEN, SPRTELE, SPRADDR, GOREMAL
32
Noteworthy components
CoHEsion Summit
Row coloring:CASE WHEN rblock.blockee_minutes_in_wait > 10 THEN rowcolor:='#FF0000'; --red WHEN rblock.blockee_minutes_in_wait > 3 THEN rowcolor:='#F4FA58'; --yellow ELSE rowcolor:='#80FF00'; --greenEND CASE;htp.tableRowOpen(cattributes=>'bgcolor="' || rowcolor || '"');
Control display of “Kill” link:if twbkwbis.f_validlink('pkg_banner_monitoring_tools.P_KillSession') then
33
Noteworthy components
CoHEsion Summit
Page refresh:twbkwbis.P_OpenDoc ('pkg_banner_monitoring_tools.P_ShowSessionsAndBlocks', refresh_delay => 30, refresh_url => OWA_UTIL.get_cgi_env ('REQUEST_PROTOCOL') || '://' || OWA_UTIL.get_cgi_env ('HTTP_HOST') || ':' || OWA_UTIL.get_cgi_env ('SCRIPT_NAME') || '/pkg_banner_monitoring_tools.P_ShowSessionsAndBlocks' );
Result = https://ui2web1.apps.uillinois.edu/BANPROD1/pkg_banner_monitoring_tools.P_ShowSessionsAndBlocks
34
Noteworthy components
CoHEsion Summit
Kill session: oracle.killsession(sid_in,serial_in,username_in,dbname,result); -- check the result CASE WHEN result LIKE '%auditname parameter not provided%' THEN P_ShowSessionsAndBlocks('NOKILL_NODB'); WHEN result LIKE '%This is a privileged ID%' THEN P_ShowSessionsAndBlocks('NOKILL_PRIVID'); WHEN result LIKE '%did not match with username%' THEN P_ShowSessionsAndBlocks('NOKILL_USRMTCH'); WHEN result LIKE '%alter system kill session%' THEN P_ShowSessionsAndBlocks('KILLED'); END CASE;
CoHEsion Summit35
Poll
36
Thank You!
Shawn Lee, University of [email protected]
Please complete the session evaluation form
Session ID #149
CoHEsion Summit