1

COMP 382: Reasoning about algorithms

Unit 9: Undecidability

[Slides adapted from Amos Israeli’s]

Limits to computation

There are problems for which there cannot be an algorithm, provably.

– Undecidable problems

There are problems for which polynomial-time algorithms are unlikely to exist

– NP-complete problems

Diagonalization

Technique for proving results about the cardinality of sets

Invented by German mathematician Georg Cantor

Cardinality

How many natural numbers are there? Infinity!

How many real numbers are there? Infinity!

Does the number of natural numbers equal the number of real numbers?

How is the size of infinite sets measured?

Cardinality

The cardinality of a set is a property marking its size.

Two sets have the same cardinality if there is a (one-to-one) correspondence between their elements

Clearly, the cardinality of A is equal to the cardinality of B.

How about the cardinality of infinite sets?

Example

7

4,3,2,1A 8,6,4,2B8642

4321

B

A

Is the cardinality of natural numbers larger than the cardinality of even natural numbers?

Question

8

Naturals and even naturals

9

nnf 2

Correspondence:

A set A is countable if it is either finite or its cardinality is equal to the cardinality of N.

“A set is countable if a list of its elements can be created”.

Countable Sets

10

“A set is countable if a list of its elements can be created”.

This list does not have to be finite, but for each natural number i, one should be able to specify the i-th element on the list.

For example, for EN the i-th element on the list is 2i .

Countable Sets

11

Is the set Q of rational numbers countable?

Question

12

Recall that each natural number is defined by a pair of natural numbers.

List rationals inan infinite rectangle.

The set of rationals is countable

14

5/54/53/52/51/5

5/4

5/3

5/2

5/1

4/43/42/41/4

4/33/32/31/3

4/23/22/21/2

4/13/12/11/1 ………

………………………

………

………………………………………………...

How can we form a list including all these numbers?

The set of rationals is countable

15

5/54/53/52/51/5

5/4

5/3

5/2

5/1

4/43/42/41/4

4/33/32/31/3

4/23/22/21/2

4/13/12/11/1 ………

………………………

………

………………………………………………...Can’t reach allnumbers

5/54/53/52/51/5

5/4

5/3

5/2

5/1

4/43/42/41/4

4/33/32/31/3

4/23/22/21/2

4/13/12/11/1

The set of rationals is countable

16

A better way

Note that some rational numbers appear more than once. For example: all numbers on the main diagonal are equal to 1, so this list is not final.

In order to compute the actual place of a given rational, we need to erase all duplicates, but this is a technicality…

The set of rationals is countable

17

The set of infinite binary sequences is not countable.

Are all sets countable?

18

Assume that there exists a list of all binary sequences. Such a list may look like this:

Uncountable Sets

19

10001

1

1

1

0

0111

0000

0011

1101 ………

………………………

………

…………………….................

But can you be sure that all sequences are in this list?

In fact, There exist infinitely many sequences that are not on the list:

Uncountable Sets

20

10001

1

1

1

0

0111

0000

0011

1101 ………

………………………

………

…………………….................

Consider for example S=0,0,1,1,0,… . The sequence S is formed so that 1st elt. Of 1st seq.

2nd elt. Of 2nd seq.

3rd elt. Of 3rd seq.

And so on …

Uncountable Sets

21

10001

1

1

1

0

0111

0000

0011

1101 ………

………………………

………

……………………...........

1S

2S

3S

Define . Obviously, for every , the i-th element of S, differs from the i-th element of the i-th sequence in the list, that is: The element on the diagonal.

Can the sequence S appear on the list?

Uncountable Sets

22

iSiS i1 iSNi

?...01100

........................

......10001

......10111

......10000

......10011

......01101

5

4

3

2

1

S

S

S

S

S

S

Uncountable Sets

23

Contradiction

Diagonalization

Back to Turing machines

Turing machine = model of a program

Inputs coded as strings over a finite input alphabet

Language = Set of strings

Intuition: A language corresponds to a computational decision problem

Example

Problem: check if a given number n is even

Language:

All inputs on which the answer is “yes”

Back to Turing machines

The collection of strings that M accepts is the language of M, denoted .

A language is Turing Recognizable if there exists a Turing machine that recognizes it.

ML

Since it is hard to tell whether a running machine is looping, we prefer machines that halt on all inputs. These machines are called deciders.

A decider that recognizes a language L is said to decide L.

A language is Turing decidable if there exists a Turing machine that decides it, and undecidable otherwise.

Turing deciders (algorithms)

27

Claim

Some Languages are not Turing-recognizable.

Proof

For any (finite) alphabet, , the set of (finite) strings , is countable. A list of all elements in is obtained by first listing strings of length 1, then 2, …, then n…

Turing unrecognizable languages

28

*

*

The set of all TM-s is also countable because every TM, , can be described by its encoding , which is a string over . So the set of TM-s corresponds to a subset of .

Note: Here we use the (unproven but correct) fact that the cardinality of a set is always not greater then the cardinality of any of its supersets.

Proof (cont.)

29

MM

*

Since each TM recognizes exactly a single language, a list of all TM-s can be used as a list of all recognizable languages.

If we show that the set of languages over is uncountable, we can deduce that at least a single language is not on the list, that is: it is not recognized by any TM.

Proof (cont.)

30

We have already seen that the set of infinite binary sequences is uncountable. Now we form a correspondence between the set of languages over and the set of infinite binary sequences to show that the set of languages is uncountable.

Proof (cont.)

31

We have already seen that the set is countable. Consider a fixed list l of all words in . A correspondence between the set of infinite binary sequences and the set of languages is formed as follows: For every infinite binary sequence S, corresponds the language:

QED

Proof (cont.)

32

*

1 where| isilSL

**

Moral of the story

There are more problems than programs

Now we’ll show an undecidable language.

The language that we prove to be undecidable is a very natural language namely the language consisting of pairs of the form where Mis a TM accepting string w:

The Halting Problem

34

wMwMATM acceptingTM a is ,

wM ,

Since this language requires to decide whether the computation of TM M halts on input w, it is often called The Halting Problem.

Is the halting problem Turing Recognizable?

The Halting Problem

35

Since this language requires to decide whether the computation of TM M halts on input w, it is often called The Halting Problem.

Claim

The halting problem is Turing Recognizable.

The Halting Problem

36

Consider a TM U that gets a pair as input and simulates the run of M on input w. If M accepts or rejects so does U. Otherwise, U loops.

Note: U recognizes ATM ,since it accepts any pair

, that is: any pair in which M accepts input w.

Proof

37

wM ,

LwM ,

TM N works as follows:

1. Mark M’s initial state and w’s initial symbol as the “current state” and “current head location”.

2. Look for M’s next transition on the description of its transition function.

3. Execute M’s transition.

A TM N that simulates an input TM N

38

4. Move M’s “current state” and “current head location” to their new places.

5. If M’s new state is a deciding state decide according to the state, otherwise – repeat stages 2-5.

Simulating an Input TM

39

So far we proved the existence of a language which is not Turing recognizable. Now we continue our quest to prove:

Theorem

The language

is undecidable.

The language is undecidable

40

TMA

wMwMATM acceptingTM a is ,

Before we start the proof let us consider two ancient questions:

Question1:

Can god create a boulder so heavy such that he (god) cannot lift?

The Language is undecidable

41

TMA

Question2:

In a small town, there is a single barber:

Over the barber’s chair there is a note saying:“I will shave you on one condition: You shall

never shave yourself.”

Who Shaves the Barber?

42

The Language is undecidable TMA

Assume, by way of contradiction, that is decidable and let H be a TM deciding .

That is

Define now another TM, D, that uses H as a subroutine as follows:

Proof

43

TMA

wMreject

wMacceptwMH

accept not does if

accepts if ,

TMA

Define now another TM new D that uses H as a subroutine as follows:

D=“On input where M is a TM:

1. Run H on input .

2. Output the opposite of H’s output namely: If H accepts reject, otherwise accept.“

Proof

44

M

MM ,

Note: What we do here is taking advantage of the two facts:

Fact1: TM M should be able to compute with any string as input.

Fact2: The encoding of M, , is a string.

Proof

45

M

Running a machine on its encoding is analogous to using a compiler for the computer language C, to compile itself (the compiler is written in C).

Proof

46

What we got now is:

Consider now the result of running D with input . What we get is:

Proof

47

M accepts if

M ejects if

Mreject

MacceptMD

r

D

D accepts if

D ejects if

Dreject

DacceptDD

r

So if D accepts, it rejects and if it rejects it accepts.

And it’s all caused by our assumption that TM H exists!

Proof

48

D accepts if

D ejects if

Dreject

DacceptDD

r

1. Define .

2. Assume that id decidable and let H be a TM deciding it.

3. Use H to build TM D that gets a string and behaves exactly opposite to H’s behavior, namely:

Proof review

49

wMwMATM acceptingTM a is ,

TMA

M accepts if

M ejects if

Mreject

MacceptMD

r

4. Run TM D on its encoding and conclude:

Contradiction.

Proof review

50

D accepts if

D ejects if

Dreject

DacceptDD

r

D

The following table describes the behavior of each machine on some machine encodings:

So where is the diagonalization?

51

acceptacceptM

M

acceptacceptacceptacceptM

acceptacceptM

MMMM

4

3

2

1

4321

This table describes the behavior of TM H. Note: TM H rejects where loops.

So where is the diagonalization?

52

rejectrejectacceptacceptM

rejectrejectrejectrejectM

acceptacceptacceptacceptM

rejectacceptrejectacceptM

MMMM

4

3

2

1

4321

iM

Now TM D is added to the table…

Proof review

53

???

4

3

2

1

4321

acceptacceptrejectrejectD

acceptrejectrejectacceptacceptM

rejectrejectrejectrejectrejectM

acceptacceptacceptacceptacceptM

acceptrejectacceptrejectacceptM

DMMMM

Use the undecidability of to prove many other languages undecidable.

Reductions

54

TMA

Show that a solution for some problem A induces a solution for problem B.

If we know that B does not have a solution, we may deduce that A is also unsolvable. In this case we say that B is reducible to A.

Reductions

55

Say you want to prove that a certain language L is undecidable.

Assume that L is decidable, and show that a decider for L, can be used to devise a decider for .

Since is undecidable, so is the language L.

Reductions

56

TMA

TMA

wMwMATM acceptingTM a is ,

Using a decider for L to construct a decider for , is called reducing to L.

Note: Once we prove that a certain language L is undecidable, we can prove that some other language, say L’ , is undecidable, by reducing L’ to L.

Reductions

57

TMA TMA

1. We know that A is undecidable.

2. We want to prove B is undecidable.

3. We assume that B is decidable and use this assumption to prove that A is decidable.

4. We conclude that B is undecidable.

Note: The reduction is from A to B.

Schematic of a Reduction

58

Consider

Theorem

is undecidable.

Proof

By reducing to

The “Real” halting problem

59

wMwMHALTTM on haltsTM that a is ,

TMHALT

TMA TMHALT

wMwMATM acceptingTM a is ,

Assume by way of contradiction that is decidable.

Recall that a decidable set has a decider R: A TM that halts on every input and either accepts or rejects, but never loops!.

We will use the assumed decider of to devise a decider for .

Intuition

60

TMHALT

TMHALT

TMA

Assume that is decidable and let R be a TM deciding it.

We give a TM S that uses R as a subroutine and decides .

Since is undecidable, this is a contradiction.

Proof

62

TMHALT

TMA

TMA

S=“On input where M is a TM:

1. Run R on input until it halts.

2. If R rejects, (i.e. M loops on w ) - reject.

(At this stage we know that R accepts, and we conclude that M halts on input w.)

3. Simulate M on w until it halts.

4. If M accepts - accept, otherwise - reject. “

Proof

63

wM ,

wM ,

Show that the following language is undecidable.

The TM Emptiness Problem

64

M AndTM a is LMMETM

The proof is by reduction from :

1. We know that is undecidable.

2. We want to prove is undecidable.

3. We assume toward a contradiction that is decidable and devise a decider for .

4. We conclude that is undecidable.

Proof Outline

65

TME

TMA

TMA

TMATME

TME

Assume that is decidable and let R be a TM deciding it.

Now we give a TM S that uses R as a subroutine and decides .

Proof

66

TME

TMA

Given an instance for , , we may try to run R on this instance. If R accepts, we know that . In particular, M does not accept w so a decider for must reject .

Proof

67

TMA

ML

wM ,

wM ,

TMA

What happens if R rejects? The only conclusion we can draw is that . What we need to know though is whether .

In order to use our decider R for , we once again modify the input machine M to obtain TM :

Proof

68

ML

MLw

TME

1M

We start with a TM satisfying .

Description of

69

1M

MLML 1

1Macceptq

rejectq

M

startq

acceptnq

rejectnq

startnq

wM

wMwML

rejects if

accepts if 1

Now we add a filter to divert all inputs but w.

Description of

70

1M

wx

wx

1Macceptq

rejectq

M

startq

acceptnq

rejectnq

startnq

wx filter

no

yes

TM has a filter that rejects all inputs excepts w, so the only input reaching M, is w.

Therefore, satisfies:

wM

wMwML

rejects if

accepts if 1

Proof

71

1M

1M

Here is a formal description of :

“On input x :

1. If - reject . 2. If - run M on w and accept if M accepts. ”

Note: M accepts w if and only if .

Proof

72

wx 1M

wx

1ML

1M

S=“On input where M is a TM:

1. Compute an encoding of TM . 2. Run R on input .

3. If R rejects - accept, otherwise - reject.

Proof

74

wM ,

1M 1M

1M

Recall that R is a decider for . If R rejects the modified machine , , hence by the specification of , , and a decider for must accept .If however R accepts, it means that , hence , and S must reject . QED

Proof

75

1M

TME

1ML

MLw1M

TMA

1ML

wM ,

MLw wM ,

76

Rice’s Theorem

We will show that for every nontrivial property P of recognizable languages is undecidable.

Consider the language LP = { <M> | M satisfies property P }

We show how to reduce to LP.

Since is undecidable, it follows that LP is also undecidable.

TMA

TMA

77

The Reduction

Our reduction algorithm takes M and w and produces a TM M’.

L(M’) has property P if and only if M accepts w.

M’ has two tapes, used for:1. Simulates another TM ML on the input to M’.

2. Simulates M on w.• neither M, ML, nor w is input to M’.

78

The Reduction (continued)

Assume that does not have property P.– If it does, consider the complement of P, which

would also be decidable.

Let L be any language with property P, and let ML be a TM that accepts L.

M’ is constructed to work as follows.

79

Design of M’

1. On the second tape, write w and then simulate M on w.

2. If M accepts w, then simulate ML on the input x to M’, which appears initially on the first tape.

3. M’ accepts its input x if and only if ML accepts x.

80

Action of M’ if M Accepts w

Simulate Mon input wx

Simulate ML

on input x

On accept

Acceptiff x isin ML

81

Design of M’ (continued)

Suppose M accepts w.

Then M’ simulates ML and therefore accepts x if and only if x is in L.

That is, L(M’) = L, L(M’) has property P.

82

Design of M’ – (3)

Suppose M does not accept w.

Then M’ never starts the simulation of ML, and never accepts its input x.

Thus, L(M’) = , and L(M’) does not have property P.

That is, M’ is not in LP.

83

Action of M’ if M Does not Accept w

Simulate Mon input wx

Never accepts, sonothing else happensand x is not accepted

84

Design of M’ – Conclusion

• Thus, the algorithm that converts M and w to M’ is a reduction of to LP.

• Thus, LP is undecidable.TMA

85

Picture of the Reduction

A realreductionalgorithm

M, wHypotheticalalgorithm for

property P

M’

Acceptiff Maccepts w

Otherwisehalt withoutaccepting

This would be an algorithmfor Lu, which doesn’t exist

86

Applications of Rice’s Theorem

• We now have any number of undecidable questions about TM’s:– Is a TM equivalent to a finite automaton?– Does a TM accept any palindromes?– Does a TM accept more than 1000 strings?– ...