AlgorithmicSpeed

BBM101- Introduction toProgramming I

Hacettepe UniversityFall2015

FuatAkal,AykutErdem,Erkut Erdem,Vahid Garousi

1Slides basedonmaterialpreparedbyE.Grimson, J.Guttag andC.Terman inMITx 6.00.1x

Measuringcomplexity

• Goalsindesigningprograms1. Itreturnsthecorrectansweronalllegalinputs2. Itperformsthecomputationefficiently

• Typically(1)ismostimportant,butsometimes(2)isalsocritical,e.g.,programsforcollisiondetection

• Evenwhen(1)ismostimportant,itisvaluabletounderstandandoptimize(2)

2

Computationalcomplexity

• Howmuchtimewillittakeaprogramtorun?• Howmuchmemorywillitneedtorun?

• Needtobalanceminimizingcomputationalcomplexitywithconceptualcomplexity– Keepcodesimpleandeasytounderstand,butwherepossibleoptimizeperformance

3

Howdowemeasurecomplexity?

• Givenafunction,wouldliketoanswer:“Howlongwillthistaketorun?”

• Couldjustrunonsomeinputandtimeit.• Problemisthatthisdependson:

1. Speedofcomputer2. SpecificsofPythonimplementation3. Valueofinput

• Avoid(1)and(2)bymeasuringtimeintermsofnumberofbasicstepsexecuted

4

Measuringbasicsteps

• Usearandomaccessmachine(RAM)asmodelofcomputation– Stepsareexecutedsequentially– Stepisanoperationthattakesconstanttime• Assignment• Comparison• Arithmeticoperation• Accessingobjectinmemory

• Forpoint(3),measuretimeintermsofsizeofinput

5

Butcomplexitymightdependonvalueofinput?

def linearSearch(L, x): for e in L:

if e==x: return True

return False

• IfxhappenstobenearfrontofL,thenreturnsTruealmostimmediately

• IfxnotinL,thencodewillhavetoexamineallelementsofL

• Needageneralwayofmeasuring

6

Casesformeasuringcomplexity• Bestcase:minimumrunningtimeoverallpossibleinputsofagivensize– ForlinearSearch – constant,i.e.independentofsizeofinputs

• Worstcase:maximumrunningtimeoverallpossibleinputsofagivensize– ForlinearSearch – linearinsizeoflist

• Average(orexpected)case:averagerunningtimeoverallpossibleinputsofagivensize

• Wewillfocusonworstcase– akindofupperboundonrunningtime

7

Example• Numberofsteps– 1 (forassignment)– 5*n (1fortest,plus2forfirstassignment,plus2forsecondassignmentinwhile;repeatedn<mesthroughwhile)

– 1 (forreturn)• 5*n+2steps• Butasngetslarge,2isirrelevant,sobasically5*nsteps

8

def fact(n): answer = 1 while n > 1:

answer *= n n -= 1

return answer

Example

• Whataboutthemultiplicativeconstant(5inthiscase)?

• Wearguethatingeneral,multiplicativeconstantsarenotrelevantwhencomparingalgorithms

9

Exampledef sqrtExhaust(x, eps):

step = eps**2 ans = 0.0 while abs(ans**2 - x) >= eps and ans <= max(x, 1):

ans += step

return ans

• Ifwecallthison100and0.0001,willtakeonebillioniterationsoftheloop– Haveroughly8stepswithineachiteration

10

Exampledef sqrtBi(x, eps):

low = 0.0high = max(1, x) ans = (high + low)/2.0 while abs(ans**2 - x) >= eps:

if ans**2 < x: low = ans

else: high = ans

ans = (high + low)/2.0 return ans

• Ifwecallthison100and0.0001,willtakethirtyiterationsoftheloop– Haveroughly10stepswithineachiteration

• 1billionor8billionversus30or300– itissizeofproblemthatmatters

11

Measuringcomplexity

• Giventhisdifferenceiniterationsthroughloop,multiplicativefactor(numberofstepswithinloop)probablyirrelevant

• Thus,wewillfocusonmeasuringthecomplexityasafunctionofinputsize–Willfocusonthelargestfactorinthisexpression–Willbemostlyconcernedwiththeworstcasescenario

12

Asymptoticnotation

• Needaformalwaytotalkaboutrelationshipbetweenrunningtimeandsizeofinputs

• Mostlyinterestedinwhathappensassizeofinputsgetsverylarge,i.e.approachesinfinity

13

Exampledef f(x):

for i in range(1000): ans = i

for i in range(x): ans += 1

for i in range(x): for j in range(x):

ans += 1

Complexityis1000+2x+2x2,ifeachlinetakesonestep

14

Example

• 1000+2x+2x2

• Ifxissmall,constanttermdominates– E.g.,x=10then1000of1220stepsareinfirstloop

• Ifxislarge,quadratictermdominates– E.g.x=1,000,000,thenfirstlooptakes0.000000005%oftime,secondlooptakes0.0001%oftime(outof2,000,002,001,000 steps)!

15

Example• Soreallyonlyneedtoconsiderthenestedloops(quadraticcomponent)

• Doesitmatterthatthisparttakes2x2 steps,asopposedtosayx2 steps?– Forourexample,ifourcomputerexecutes100millionstepspersecond,differenceis5.5hoursversus2.25hours

– Ontheotherhandifwecanfindalinearalgorithm,thiswouldruninafractionofasecond

– Somultiplicativefactorsprobablynotcrucial,butorderofgrowthiscrucial

16

Rulesofthumbforcomplexity

• Asymptoticcomplexity– Describerunningtimeintermsofnumberofbasicsteps

– Ifrunningtimeissumofmultipleterms,keeponewiththelargestgrowthrate

– Ifremainingtermisaproduct,dropanymultiplicativeconstants

• Use“BigO”notation(akaOmicron)– Givesanupperboundonasymptoticgrowthofafunction

17

Complexityclasses

• O(1)denotesconstantrunningtime• O(logn)denoteslogarithmicrunningtime• O(n)denoteslinearrunningtime• O(nlogn)denoteslog-linearrunningtime• O(nc)denotespolynomialrunningtime(cisaconstant)

• O(cn)denotesexponentialrunningtime(cisaconstantbeingraisedtoapowerbasedonsizeofinput)

18

Constantcomplexity

• Complexityindependentofinputs• Veryfewinterestingalgorithmsinthisclass,butcanoftenhavepiecesthatfitthisclass

• Canhaveloopsorrecursivecalls,butnumberofiterationsorcallsindependentofsizeofinput

19

Logarithmiccomplexity

• Complexitygrowsaslogofsizeofoneofitsinputs

• Example:– Bisectionsearch– Binarysearchofalist

20

Logarithmiccomplexity

21

def intToStr(i): digits = '0123456789' if i == 0:

return '0' result = '' while i > 0:

result = digits[i%10] + result i = i/10

return result

Logarithmiccomplexity• Onlyhavetolookatloopasnofunctioncalls

• Withinwhileloopconstantnumberofsteps

• Howmanytimesthroughloop?– Howmanytimescanonedividei by10?

– O(log(i))

22

def intToStr(i): digits = '0123456789' if i == 0:

return '0' result = '' while i > 0:

result = digits[i%10] + result

i = i/10 return result

Linearcomplexity• Searchingalistinordertoseeifanelementispresent

• Addcharactersofastring,assumedtobecomposedofdecimaldigits

def addDigits(s): val = 0 for c in s:

val += int(c)

return val

• O(len(s))23

Linearcomplexity• Complexitycandependonnumberofrecursivecalls

def fact(n): if n == 1:

return 1 else:

return n*fact(n-1)

• Numberofrecursivecalls?– Fact(n),thenfact(n-1),etc.untilgettofact(1)– Complexityofeachcallisconstant– O(n) 24

Log-linearcomplexity

• Manypracticalalgorithmsarelog-linear• Verycommonlyusedlog-linearalgorithmismergesort

• Willreturntothis

25

Polynomialcomplexity

• Mostcommonpolynomialalgorithmsarequadratic,i.e.,complexitygrowswithsquareofsizeofinput

• Commonlyoccurswhenwehavenestedloopsorrecursivefunctioncalls

26

Quadraticcomplexity

27

def isSubset(L1, L2): for e1 in L1:

matched = False for e2 in L2:

if e1 == e2: matched = True break

if not matched: return False

return True

Quadraticcomplexity

• Outerloopexecutedlen(L1)times

• Eachiterationwillexecuteinnerloopuptolen(L2)times

• O(len(L1)*len(L2))• WorstcasewhenL1

andL2samelength,noneofelementsofL1inL2

• O(len(L1)2)28

def isSubset(L1, L2): for e1 in L1:

matched = False for e2 in L2:

if e1 == e2: matched = True break

if not matched: return False

return True

Quadraticcomplexity

Findintersectionoftwolists,returnalistwitheachelementappearingonlyonce

29

def intersect(L1, L2): tmp = [] for e1 in L1:

for e2 in L2: if e1 == e2:

tmp.append(e1) res = [] for e in tmp:

if not(e in res): res.append(e)

return res

Quadraticcomplexity

• Firstnestedlooptakeslen(L1)*len(L2)steps

• Secondlooptakesatmostlen(L1)steps

• Lattertermoverwhelmedbyformerterm

• O(len(L1)*len(L2))

30

def intersect(L1, L2): tmp = [] for e1 in L1:

for e2 in L2: if e1 == e2:

tmp.append(e1) res = [] for e in tmp:

if not(e in res): res.append(e)

return res

Exponentialcomplexity

• Recursivefunctionswheremorethanonerecursivecallforeachsizeofproblem– TowersofHanoi

• Manyimportantproblemsareinherentlyexponential– Unfortunate,ascostcanbehigh–Willleadustoconsiderapproximatesolutionsmorequickly

31

Exponentialcomplexity

32

def genSubsets(L): res = [] if len(L) == 0:

return [[]] #list of empty list smaller = genSubsets(L[:-1]) # get all subsets without last element extra = L[-1:] # create a list of just last element new = [] for small in smaller:

new.append(small+extra) # for all smaller solutions, add one with last element return smaller+new# combine those with last element and those without

Exponentialcomplexity

• Assumingappendisconstanttime

• Timeincludestimetosolvesmallerproblem,plustimeneededtomakeacopyofallelementsinsmallerproblem

33

def genSubsets(L): res = [] if len(L) == 0:

return [[]] smaller = genSubsets(L[:-1]) extra = L[-1:] new = [] for small in smaller:

new.append(small+extra) return smaller+new

Exponentialcomplexity

• Butimportanttothinkaboutsizeofsmaller

• Knowthatforasetofsizekthereare2kcases

• Sotosolveneed2n-1 +2n-2 +...+20 steps

• MathtellsusthisisO(2n)

34

def genSubsets(L): res = [] if len(L) == 0:

return [[]] smaller = genSubsets(L[:-1]) extra = L[-1:] new = [] for small in smaller:

new.append(small+extra) return smaller+new

Complexityclasses

• O(1)denotesconstantrunningtime• O(logn)denoteslogarithmicrunningtime• O(n)denoteslinearrunningtime• O(nlogn)denoteslog-linearrunningtime• O(nc)denotespolynomialrunningtime(cisaconstant)

• O(cn)denotesexponentialrunningtime(cisaconstantbeingraisedtoapowerbasedonsizeofinput)

35

Comparingcomplexities

• Sodoesitreallymatterifourcodeisofaparticularclassofcomplexity?

• Dependsonsizeofproblem,butforlargescaleproblems,complexityofworstcasemakesadifference

36

ConstantversusLogarithmic

37

Constant*versus*logarithmic*

Observations

• Alogarithmicalgorithmisoftenalmostasgoodasaconstanttimealgorithm

• Logarithmiccostsgrowveryslowly

38

LogarithmicversusLinear

39

Logarithmic*versus*Linear*

Observations

• Logarithmicclearlybetterforlargescaleproblemsthanlinear

• Doesnotimplylinearisbad,however

40

LinearversusLog-linear

41

Linear*versus*LogFlinear*

Observations

• Whilelog(n)maygrowslowly,whenmultipliedbyalinearfactor,growthismuchmorerapidthanpurelinear

• O(nlogn)algorithmsarestillveryvaluable

42

Log-linearversusQuadratic

43

LogFlinear*versus*Quadra/c*

Observations

• Quadraticisoftenaproblem,however.• Someproblemsinherentlyquadraticbutifpossiblealwaysbettertolookformoreefficientsolutions

44

QuadraticversusExponential• Exponentialalgorithmsveryexpensive– Rightplotisonalogscale,sinceleftplotalmostinvisiblegivenhowrapidlyexponentialgrows

• Exponentialgenerallynotofuseexceptforsmallproblems

45

Quadra/c*versus*Exponen/al*

•  Exponen/al*algorithms*very*expensive*– Right*plot*is*on*a*log*scale,*since*leD*plot*almost*invisible*given*how*rapidly*exponen/al*grows*

•  Exponen/al*generally*not*of*use*except*for*small*problems**