Tech Talks_04.07.15_Session 2_Danail Branekov_Avoiding And Diagnosing Deadlocks In Java...
-
Upload
epamsystemsbulgaria -
Category
Documents
-
view
58 -
download
1
Transcript of Tech Talks_04.07.15_Session 2_Danail Branekov_Avoiding And Diagnosing Deadlocks In Java...
Avoiding and Diagnosing
DEADLOCKS
Danail Branekov
July 4, 2015
2CONFIDENTIAL
CREDITS
Dr Heinz Kabutz
• Born in Cape Town, South Africa, now lives in Greece
• Has a PhD in Computer Science
• Created the Java Specialists’ Newsletter
• One of the first Sun Java Champions
• Carries out Java training courses all over the world
• Regular speaker at all the major Java conferences
This presentation is based on materials from Heinz’ Extreme Java – Concurrency and
Performance training with his explicit permission
3CONFIDENTIAL
FROM THE CLASSIC
• Classic problem is that of the dining philosophers which we change to drinking
philosophers
• This is where the word symposium comes from:
• sym – together, such as in symphony
• poto - drink
• Ancient Greek philosophers used to get together to drink & think
• In our example a philosopher needs two glasses to drink
• First he takes the right one, then the left one
• When he finishes drinking he returns them and carries on thinking
4CONFIDENTIAL
WORST CASE SCENARIO
Table is ready, all philosophers are thinking
1
5 2
4 3
5CONFIDENTIAL
WORST CASE SCENARIO
Philosopher 5 wants to drink, takes right cup
1
5 2
4 3
6CONFIDENTIAL
WORST CASE SCENARIO
Philosopher 1 wants to drink, takes right cup
5 2
4 3
1
7CONFIDENTIAL
2
WORST CASE SCENARIO
Philosopher 2 wants to drink, takes right cup
5
4 3
1
8CONFIDENTIAL
3
2
WORST CASE SCENARIO
Philosopher 3 wants to drink, takes right cup
5
4
1
9CONFIDENTIAL
4 3
2
WORST CASE SCENARIO
Philosopher 4 wants to drink, takes right cup
5
1
10CONFIDENTIAL
4 3
2
WORST CASE SCENARIO
DEADLOCK
• All philosophers are waiting
for their left cups but they
will never become available
5
1
11CONFIDENTIAL
GLOBAL ORDER WITH BOOZING
PHILOSOPHERS
We can solve the deadlock with the drinking philosophers by requiring that locks are
always acquired in a set order
• For example, we can make a rule that philosophers always first take the cup with the
largest number
• And return the cup with the lowest number first
12CONFIDENTIAL
GLOBAL LOCK ORDERING
Table is ready, all philosophers are thinking
1
5 2
4 3
1 2
3
4
5
13CONFIDENTIAL
5
GLOBAL LOCK ORDERING
Philosopher 5 takes cup 5
• remember our rule!1
2
4 3
1 2
3
4
5
14CONFIDENTIAL
1
5
GLOBAL LOCK ORDERING
Philosopher 1 takes cup 2
• remember our rule!
2
4 3
1
3
4
5
2
15CONFIDENTIAL
2
1
5
GLOBAL LOCK ORDERING
Philosopher 2 takes cup 3
4 3
1
4
5
2
3
16CONFIDENTIAL
3
2
1
5
GLOBAL LOCK ORDERING
Philosopher 3 takes cup 4
4
1
5
2
3
4
17CONFIDENTIAL
3
2
1
5
GLOBAL LOCK ORDERING
Philosopher 1 takes cup 1 - Drinking
4
5
2
3
4
1
18CONFIDENTIAL
1
3
25
GLOBAL LOCK ORDERING
Philosopher 1 finished drinking, returns cup 1
• Cups are returned in the opposite
order to what they are
acquired
4
5
2
3
4
1
19CONFIDENTIAL
1
3
25
GLOBAL LOCK ORDERING
Philosopher 5 takes cup 1 - Drinking
4
5
2
3
4
1
20CONFIDENTIAL
5
1
3
2
GLOBAL LOCK ORDERING
Philosopher 5 returns cup 1
4
5
2
3
4
1
21CONFIDENTIAL
5
1
3
2
GLOBAL LOCK ORDERING
Philosopher 1 returns cup 2
4
5
2
3
4
1
22CONFIDENTIAL
5
1
3
2
GLOBAL LOCK ORDERING
Philosopher 2 takes cup 2 - Drinking
4
5
2
3
4
1
23CONFIDENTIAL
5
1
3
2
GLOBAL LOCK ORDERING
Philosopher 5 returns cup 5
4
5
2
3
4
1
24CONFIDENTIAL
4
5
1
3
2
GLOBAL LOCK ORDERING
Philosopher 4 takes cup 5
5
2
3
4
1
25CONFIDENTIAL
2
4
5
1
3
GLOBAL LOCK ORDERING
Philosopher 2 returns cup 2
5
2
3
4
1
26CONFIDENTIAL
2
4
5
1
3
GLOBAL LOCK ORDERING
Philosopher 2 returns cup 3
5
2
3
4
1
27CONFIDENTIAL
2
4
5
1
3
GLOBAL LOCK ORDERING
Philosopher 3 takes cup 3 - Drinking
5
2
3
4
1
28CONFIDENTIAL
3
2
4
5
1
GLOBAL LOCK ORDERING
Philosopher 3 returns cup 3
5
2
3
4
1
29CONFIDENTIAL
3
2
4
5
1
GLOBAL LOCK ORDERING
Philosopher 3 returns cup 4
5
2
3
4
1
30CONFIDENTIAL
3
2
4
5
1
GLOBAL LOCK ORDERING
Philosopher 4 takes cup 4 - Drinking
5
2
3
4
1
31CONFIDENTIAL
4 3
25
1
GLOBAL LOCK ORDERING
Philosopher 4 returns cup 4
5
2
3
4
1
32CONFIDENTIAL
4 3
25
1
GLOBAL LOCK ORDERING
Philosopher 4 returns cup 5
5
2
3
4
1
33CONFIDENTIAL
LOCK ORDERING AS JAVA CODE
public class Symposium {
private final Cup[] cups;
private final Thinker[] thinkers;
public Symposium(int delegates) {
cups = new Cup[delegates];
thinkers = new Thinker[delegates];
for (int i = 0; i < cups.length; i++) {
cups[i] = new Cup();
}
for (int i = 0; i < delegates; i++) {
Cup right = cups[i];
Cup left = cups[(i + 1) % delegates];
thinkers[i] = new Thinker(i, left, right);
}
}
34CONFIDENTIAL
LOCK ORDERING AS JAVA CODE
public class Symposium {
. . .
public ThinkerStatus run() throws InterruptedException {
ExecutorService exec = Executors.newCachedThreadPool();
for (Thinker thinker : thinkers) {
exec.submit(thinker);
}
. . .
}
}
35CONFIDENTIAL
LOCK ORDERING AS JAVA CODE
public class Thinker implements Callable<ThinkerStatus> {
private final int id;
private final Cup smaller, bigger;
private int drinks = 0;
public Thinker(int id, Cup right, Cup left) {
this.id = id;
this.bigger = right.compareTo(left) > 0 ?
right : left;
this.smaller = this.bigger == right ?
left : right;}
public ThinkerStatus call() throws Exception {
for (int i = 0; i < 1000; i++) {drink(); think();}return drinks == 1000 ? ThinkerStatus.HAPPY_THINKER :
ThinkerStatus.UNHAPPY_THINKER;
}
. . .
public void think() { . . . }
}
36CONFIDENTIAL
LOCK ORDERING AS JAVA CODE
public class Thinker implements Callable<ThinkerStatus> {
private final Cup smaller, bigger;
. . .
public Thinker(int id, Cup right, Cup left) {
this.bigger = right.compareTo(left) > 0 ?
right : left;
this.smaller = this.bigger == right ?
left : right;
}
public void drink() {
synchronized (bigger) {
synchronized (smaller) {
drinking();
}
}
}
private void drinking() { drinks++; }
}
37CONFIDENTIAL
LOCK ORDERING AS JAVA CODE (VER 2)
public class Thinker implements Callable<ThinkerStatus> {
private static final Object TIE_LOCK = new Object();
. . .
public void drink() {
long leftHashCode = System.identityHashCode(left);
long rightHashCode = System.identityHashCode(right);
if (leftHashCode > rightHashCode) {
synchronized (left) {
synchronized (right) { drinking(); }
}
} else if (leftHashCode < rightHashCode) {
synchronized (right) {
synchronized (left) { drinking(); }
}
} else {
synchronized (TIE_LOCK) {
synchronized (right) {
synchronized (left) { drinking(); }
}
}
}
38CONFIDENTIAL
DEADLOCKING COOPERATING OBJECTS
In this example, the deadlock is more subtle
• Taxi is an individual taxi with a location and destination
• Dispatcher represents a fleet of taxis, capable of providing an image representing the
current location of taxis
Spot the deadlock
39CONFIDENTIAL
TAXI, REPRESENTING AN INDIVIDUAL VEHICLE
public class Taxi {
private Point location, destination;
private Dispatcher dispatcher;
public void setDispatcher(Dispatcher dispatcher) {
this.dispatcher = dispatcher;
}
public void setDestination(Point destination) {
this.destination = destination;
}
public synchronized Point getLocation() {
return location;
}
public synchronized void setLocation(Point location) {
this.location = location;
if (location.equals(destination))
dispatcher.notifyAvailable(this);
}
}
}
40CONFIDENTIAL
DISPATCHER, MANAGING A FLEET OF TAXIS
public class Dispatcher {
private final Set<Taxi> taxis;
private final Set<Taxi> availableTaxis;
public Dispatcher(Taxi... taxis) {
this.taxis = new HashSet<Taxi>(Arrays.asList(taxis));
this.availableTaxis = new
HashSet<Taxi>(Arrays.asList(taxis));
}
public synchronized void notifyAvailable(Taxi taxi) {
availableTaxis.add(taxi);
}
public synchronized Image getImage() {
Image image = new Image();
taxis.forEach(taxi -> image.drawMarker(taxi.getLocation()));
return image;
}
}
41CONFIDENTIAL
HOW TO DEADLOCK THE TAXI INDUSTRY
AsetLocation()
synchronized(taxi)
dispatcher.notifyAvailable(taxi)
synchronized(dispatcher)
BgetImage()
synchronized(dispatcher)
taxi.getLocation()
synchronized(taxi)
BLOCKED
BLOCKED
42CONFIDENTIAL
OPEN CALLS
• Calling an alien method with a lock held is difficult to analyze and therefore risky
• Both Taxi and Dispatcher violate this rule
• Calling a method with no locks held is called an open call
• Makes it much easier to reason about liveness
43CONFIDENTIAL
REFACTORED Taxi.setLocation()
• We should not call alien methods whilst holding locks
• Here we split the method up into parts that need the lock and those that call alien
methods
public void setLocation(Point location) {
boolean reachedDestination;
synchronized (this) {
this.location = location;
reachedDestination = location.equals(destination);
}
if (reachedDestination) {
dispatcher.notifyAvailable(this);
}
}
44CONFIDENTIAL
REFACTORED Dispatcher.getImage()
• We make a copy of the taxis set to prevent race conditions
public Image getImage() {
Set<Taxi> copy;
synchronized (this) {
copy = new HashSet<>(taxis);
}
Image image = new Image();
copy.forEach(taxi -> image.drawMarker(taxi.getLocation()));
return image;
}
45CONFIDENTIAL
DEADLOCK SOLVED
AsetLocation()
synchronized(taxi)
dispatcher.notifyAvailable(taxi)
synchronized(dispatcher)
BgetImage()
synchronized(dispatcher)
taxi.getLocation()
synchronized(taxi)
46CONFIDENTIAL
ANOTHER SOLUTION
• An alternative solution would be to make the Taxi.location field volatile which allows
making the getLocation method not synchronized
public class Taxi {
private volatile Point location;
. . .
public Point getLocation() {
return location;
}
}
AsetLocation()
synchronized(taxi)
dispatcher.notifyAvailable(taxi)
synchronized(dispatcher)
BgetImage()
synchronized(dispatcher)
taxi.getLocation()
synchronized(taxi)
47CONFIDENTIAL
BENEFITS OF OPEN CALLS
• Strive to use open calls throughout your program
• Programs that rely on open calls are far easier to analyze for deadlock-freedom than
those that allow calls to alien methods with locks held
• Alien method calls with lock held are probably the biggest cause of deadlocks in the
field
48CONFIDENTIAL
OPEN CALL IN Vector
• In Sun Java 6 Vector.writeObject() is synchronized
private synchronized void writeObject(java.io.ObjectOutputStream s)
throws java.io.IOException {
s.defaultWriteObject();
}
• The reason to have the method synchronized is to provide thread safety during writing
• However, since it calls the alien s.defaultWriteObject() it can deadlock
• http://www.javaspecialists.eu/archive/Issue184.html
49CONFIDENTIAL
OPEN CALL IN Vector
IBM avoids this problem with an open call
private void writeObject(ObjectOutputStream stream) throws IOException {
Vector<E> cloned = null;
// this specially fix is for a special dead-lock in customer
// program: two vectors refer each other may meet dead-lock in
// synchronized serialization. Refer CMVC-103316.1
synchronized (this) {
try {
cloned = (Vector<E>) super.clone();
cloned.elementData = elementData.clone();
} catch (CloneNotSupportedException e) {
// no deep clone, ignore the exception
}
}
cloned.writeObjectImpl(stream);
}
private void writeObjectImpl(ObjectOutputStream stream) throws IOException {
stream.defaultWriteObject();}
50CONFIDENTIAL
OPEN CALL IN Vector
OpenJDK 7 also uses an open call
private void writeObject(ObjectOutputStream s) throws IOException {
final ObjectOutputStream.PutField fields = s.putFields();
final Object[] data;
synchronized (this) {
fields.put("capacityIncrement", capacityIncrement);
fields.put("elementCount", elementCount);
data = elementData.clone();
}
fields.put("elementData", data);
s.writeFields();
}
51CONFIDENTIAL
RESOURCE DEADLOCKS
• We can also cause deadlocks waiting for resources
• For example, you may have two DB connection pools (D1 and D2)
• Some tasks might require connection to both databases
• Thus thread A might hold a semaphore for D1 and wait for D2, while thread B
might hold a semaphore for D1 and be waiting for D2
public class DatabasePool {
private final Semaphore connections;
public DatabasePool(int connections) {
this.connections = new Semaphore(connections);
}
public void connect() {
connections.acquireUninterruptibly();System.out.println("connect");
}
public void disconnect() {
System.out.println("disconnect");connections.release();
}
}
• NOTE: Thread dump and ThreadMXBean do not show this as deadlock!
52CONFIDENTIAL
ANTI-DEADLOCK BEST PRACTICES
• If you only ever acquire one lock, you cannot get a lock-ordering lock
• This is the easiest way to avoid deadlocks but it is not always practical
• If you need to acquire multiple locks, include lock ordering in your design
• It is important to specify and document possible lock sequences
• Identify where multiple locks could be acquired
• Do a global analysis to ensure that lock ordering is consistent (could be
extremely difficult on large programs)
• Use open calls when possible
• Do not call alien methods whilst holding a lock
53CONFIDENTIAL
UNIT TESTING FOR LOCK ORDERING
DEADLOCKS
• Code typically has to be called many times before a deadlock occurs
• How many times do you need to call it to prove that there is no deadlock?
• Nondeterministic unit tests are bad, a unit test should either always fail, or always
pass
54CONFIDENTIAL
UNIT TESTING FOR LOCK ORDERING
DEADLOCKS
Consider the following example
public class Bank {
public boolean transferMoney(Account from,
Account to, DollarAmount amount) {
synchronized (from) {
synchronized (to) {
return doActualTransfer(from, to, amount);
}
}
}
}
In the transferMoney method a deadlock occurs if after the first lock (from) is granted,
the first thread is swapped out and another thread requests the second lock (to)
55CONFIDENTIAL
UNIT TESTING FOR LOCK ORDERING
DEADLOCKS
• We can force the deadlock by sleeping a short while after requesting the first lock
public class Bank {
public boolean transferMoney(Account from,
Account to, DollarAmount amount) {
synchronized (from) {
sleepAWhileForTesting();
synchronized (to) {
return doActualTransfer(from, to, amount);
}
}
}
protected void sleepAWhileForTesting() {}
}
• The empty sleepAWhileForTesting method will be optimized away by the HotSpot
compiler
56CONFIDENTIAL
UNIT TESTING FOR LOCK ORDERING
DEADLOCKS
• In our unit test we extend the class and use it instead
public class SlowBank extends Bank {
private final long timeout;
private final TimeUnit unit;
public SlowBank(long timeout, TimeUnit unit) {
this.timeout = timeout; this.unit = unit;
}
@Override
protected void sleepAWhileForTesting() {
try {
unit.sleep(timeout);
} catch (InterruptedException e) { . . . }
}
}
• We can then implement a parallel task which checks for thread deadlocks and fail the
test in case of one
57CONFIDENTIAL
FINDING DEADLOCKED THREADS
• ThreadMXBean has two methods for finding deadlocks
• findMonitorDeadlockedThreads()
• Include only “monitor” locks, i.e. synchronized
• The only way to find deadlock in Java 5
• findDeadlockedThreads()
• Include “monitor” and “owned” (Java 5) locks
• Preferred method to test for deadlocks
• Does not find deadlocks between semaphores
• Both methods are designed for troubleshooting use but not for synchronization control
• Finding deadlocked threads might be an expensive operation
• http://www.javaspecialists.eu/archive/Issue130.html
58CONFIDENTIAL
FINDING DEADLOCKED THREADS
private static final ThreadMXBean tmb =
ManagementFactory.getThreadMXBean();
public Collection<Thread> findDeadlockedThreads() {
long[] ids = tmb.findDeadlockedThreads();
if (ids == null) { return Collections.emptyList();}
return findThreadsByIds(ids);
}
. . .
private Thread findThreadById(long threadId) {
for (Thread thread : Thread.getAllStackTraces().keySet()) {
if (thread.getId() == threadId)
return thread;
}
return null;
}
59CONFIDENTIAL
DEADLOCK ANALYSIS WITH THREAD DUMPS
• We can also analyze deadlocks via looking at the thread dump
• Thread dump can be caused in many ways:
• Ctrl-Break on Windows or Ctrl-\ on Unix
• Invoking kill -3 on the process id
• Calling jstack on the process id (only shows deadlocks since Java 6)
• It is useful to have unique thread names
60CONFIDENTIAL
DEADLOCK ANALYSIS WITH THREAD DUMPS
Found one Java-level deadlock:
=============================
"pool-1-thread-2":
waiting to lock monitor 0x0000000002ae5bb8 (object 0x000000076b814b08, a
taxi.deadlock.Dispatcher),
which is held by "pool-1-thread-1"
"pool-1-thread-1":
waiting to lock monitor 0x000000001e08c458 (object 0x000000076b812b88, a
taxi.deadlock.Taxi),
which is held by "pool-1-thread-2"
61CONFIDENTIAL
DEADLOCK ANALYSIS WITH THREAD DUMPS
Java stack information for the threads listed above:
===================================================
"pool-1-thread-2":
at taxi.deadlock.Dispatcher.notifyAvailable(Dispatcher.java:17)
- waiting to lock <0x000000076b814b08> (a taxi.deadlock.Dispatcher)
at taxi.deadlock.Taxi.setLocation(Taxi.java:30)
- locked <0x000000076b812b88> (a taxi.deadlock.Taxi)
at taxi.deadlock.TaxiDriver.run(TaxiDriver.java:25)
...
"pool-1-thread-1":
at taxi.deadlock.Taxi.getLocation(Taxi.java:19)
- waiting to lock <0x000000076b812b88> (a taxi.deadlock.Taxi)
at taxi.deadlock.Dispatcher.lambda$0(Dispatcher.java:22)
at taxi.deadlock.Dispatcher$$Lambda$1/710209934.accept(Unknown Source)
at java.lang.Iterable.forEach(Iterable.java:75)
at taxi.deadlock.Dispatcher.getImage(Dispatcher.java:22)
- locked <0x000000076b814b08> (a taxi.deadlock.Dispatcher)
at taxi.deadlock.TaxiObserver.run(TaxiObserver.java:14)
...
Found 1 deadlock.
62CONFIDENTIAL
DEADLOCK ANALYSIS WITH THREAD DUMPS
ThreadMXBean (and JConsole) do not detect lock order deadlocks caused by acquiring
semaphoresSemaphore s1 = new Semaphore(1);
Semaphore s2 = new Semaphore(1);
ExecutorService execService = Executors.newCachedThreadPool();
execService.submit(new TestTask(s1, s2));
execService.submit(new TestTask(s2, s1));
static class TestTask implements Runnable {
public TestTask(Semaphore first, Semaphore second) {
this.first = first;
this.second = second;
}
public void run() {
try {
first.acquire();
second.acquire();
. . .
second.release();
first.release();
} catch (InterruptedException e) { . . . }
}
}
63CONFIDENTIAL
DEADLOCK ANALYSIS WITH THREAD DUMPS
ThreadMXBean (and JConsole) do not detect lock order deadlocks caused by acquiring
semaphores
"pool-1-thread-1" #15 prio=5 os_prio=0 tid=0x000000001ddb2800 nid=0x1f78 waiting on
condition [0x000000001e69e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076b9637a8> (a
java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Abstract
QueuedSynchronizer.java:836)
. . .
64CONFIDENTIAL
DEADLOCK RECOVERY
Can I recover from a deadlock?
• DISCLAIMER: Always prefer deadlock prevention via implementation best practices
over recovering from deadlocks which already took place. Recovering from deadlocks
should be only used as an extreme measure
• NO, if you deadlocked with synchronized
• synchronized goes into BLOCKED state
• YES, with ReentrantLock
• ReentrantLock.lock() and ReentrantLock.lockInterruptibly() go into WAITING
state
• Deadlocked threads must make sure that they release locks held when
interrupted or stopped
• We can implement a DeadlockArbitrator which interrupts or stops one of the
threads
65CONFIDENTIAL
DEADLOCK RECOVERY VIA INTERRUPT
ReentrantLock first = new ReentrantLock();
ReentrantLock second = new ReentrantLock();
private class TestTask implements Runnable {
public void run() {
try {
first.lockInterruptibly();try {
// do some work
second.lockInterruptibly();try {
// do more work
} finally {
second.unlock();
}
} finally {
first.unlock();
}
} catch (InterruptedException e) { . . . }
}
66CONFIDENTIAL
DEADLOCK RECOVERY VIA INTERRUPT
public class ThreadInterruptingDeadlockArbitrator {
public void tryResolveDeadlock() throws InterruptedException {
Thread t = findDeadlockedThread(); // via ThreadMXBean
if (t != null) {
t.interrupt();
}
}
}
67CONFIDENTIAL
DEADLOCK RECOVERY VIA STOP
ReentrantLock first = new ReentrantLock();
ReentrantLock second = new ReentrantLock();
private class TestTask implements Runnable {
public void run() {
first.lock();try {
// do some work
second.lock();try {
// do more work
} finally {
second.unlock();
}
} finally {
first.unlock();
}
}
}
68CONFIDENTIAL
DEADLOCK RECOVERY VIA STOP
public class ThreadInterruptingDeadlockArbitrator {
public void tryResolveDeadlock() throws InterruptedException {
Thread t = findDeadlockedThread(); // via ThreadMXBean
if (t != null) {
t.stop();
}
}
}
69CONFIDENTIAL
LINKS
• Java Concurrency in Practice by Brian Goetz
• The Java Specialists’ Newsletter
• Sample source code
70CONFIDENTIAL
THANK YOU!