02_PriorityInversion

7/31/2019 02_PriorityInversion

1/22

Click to edit Master subtitle style

8/5/12

PRIORITY INVERSIONPROBLEMRAVINDRA V. JOSHI


2/22

8/5/12

MARS PATH FINDER


3/22

8/5/12

Description of the

Problem occurredPath Finder was getting reset every now andthen

System appeared to be unstableCrucial data was getting lost as system was

getting reset

No clue what was the problem, as there wasno test report suggested this behavior!!!

How to analyze? How to debug?? How tocorrect???


4/22

8/5/12

Analysis Of the ProblemInformation bus High Priority Task

Data Gathering Task Low Priority Task

Information Bus Task was getting resetcontinuously

Finally got zeroed-on to Priority InversionProblem


5/22


6/22

8/5/12

TASKS AND RESOURCESA typical system contains a set of tasks and

set of resources


7/22

8/5/12

SCHEDULING ART OF

ASSIGNING RESOURCESTO TASKSFIFOShortest Job First

Priority based scheduling

Round Robin

Pre-emptive and Non-PreEmptive ( Co-

operative )

Longest Waiting First


8/22

8/5/12

Priority Based

Pre-Emptive Scheduling


9/22

8/5/12

Priority Based



10/22

8/5/12

Priority Based



11/22

8/5/12

Priority Based



12/22

8/5/12

Locking a Resource


13/22

8/5/12

DESCRIPTION OF

PRIORITY INVERSIONConsider 3 Tasks with Priority J


14/22

8/5/12

Currently J is scheduled and it locks JOKER

DESCRIPTION OF

PRIORITY INVERSION


15/22

8/5/12

Now Q becomes Ready and Pre Empts J as itdoes not require JOKER

DESCRIPTION OF

PRIORITY INVERSION


16/22

8/5/12

Q got scheduled though K with higher priorityis waiting. This is PRIORITY INVERSION.

DESCRIPTION OF

PRIORITY INVERSION


17/22


18/22

8/5/12

Debugging and

Correcting the problemProblem was tried for reproduction on ground

by collected Traces and Logs

At last, the problem got reproduced wheneverybody gave up and went home exceptone engineer who finally saw the reproduction

It turned out that VxWorks( the operating

system inside Path Finder ) had implementeda solution ( with Priority Inheritance ) but thatfunction was called with parameter false

A short patch overwriting the parameter was

beamed to Mars and Program was Corrected


19/22

8/5/12

LESSONS LEARNT

Never leave distant machine in Un-debuggable method. Some format of Remotedebugging MUST be present

Exercise the options conservatively. Whencost of using or not using Priority

Inheritance is not extraordinarily large, usePriority Inheritance

Do not give up hope


20/22

8/5/12

Unsung Heroes

But who had found that such problem canoccur? It was quiet three Academicians fromCarnegie Mellon University

Lui Sha

John Lehoczky and

Raj Rajkumar

Standing Applause


21/22

8/5/12

Questions??

THANK YOU


22/22

8/5/12

WATCHDOG The

Ultimate SaviorBasic Principle is Never to hang

If required, Automatically Resetting is

preferredIf it finds, that the system is HUNG, it will

reset the system

02_PriorityInversion

Documents

Transcript of 02_PriorityInversion