Lecture 4 SQL Adv II - GitHub Pages · •I.e. If we join relations A and B on a.X= b.X, and there...

Lecture4:AdvancedSQL– PartII

Announcements!

1. ProblemSet#1isreleased!• Wewilldiscusssomeofthequestionsattheendofthislecture

2. Projectgroupassignments• Doeseverybodyhaveateam?

3. Askquestions,Gotoofficehours,EngageonPiazza

2

Lecture4

Lecture4:AdvancedSQL– PartII

Lecture4

Today’sLecture

1. Aggregation&GROUPBY• ACTIVITY:FancySQLPartI

2. AdvanceSQL-izing• ACTIVITY:FancySQLPartII

3. ProblemSet#1Overview

4

Lecture4

1.Aggregation&GROUPBY

5

Lecture4>Section1

Whatyouwilllearnaboutinthissection

1. Aggregationoperators

2. GROUPBY

3. GROUPBY:withHAVING,semantics

4. ACTIVITY:FancySQLPt.I

6

Lecture4>Section1

7

Aggregation

SELECT COUNT(*)FROM ProductWHERE year > 1995

ExceptCOUNT,allaggregationsapplytoasingleattribute

SELECT AVG(price)FROM ProductWHERE maker = “Toyota”

• SQLsupportsseveralaggregation operations:• SUM,COUNT,MIN,MAX,AVG

Lecture4>Section1>Aggregation

8

• COUNTappliestoduplicates,unlessotherwisestated

SELECT COUNT(category) FROM ProductWHERE year > 1995

Note:SameasCOUNT(*).Why?

Weprobablywant:

SELECT COUNT(DISTINCT category)FROM ProductWHERE year > 1995

Aggregation:COUNT


9

Purchase(product, date, price, quantity)

MoreExamples

SELECT SUM(price * quantity)FROM Purchase

SELECT SUM(price * quantity)FROM PurchaseWHERE product = ‘bagel’

Whatdothesemean?


10

SimpleAggregationsPurchaseProduct Date Price Quantitybagel 10/21 1 20banana 10/3 0.5 10banana 10/10 1 10bagel 10/25 1.50 20

SELECT SUM(price * quantity)FROM PurchaseWHERE product = ‘bagel’

50(=1*20+1.50*20)


11

GroupingandAggregation

SELECT product,SUM(price * quantity) AS TotalSales

FROM PurchaseWHERE date > ‘10/1/2005’GROUP BY product

Let’sseewhatthismeans…

Findtotalsalesafter10/1/2005perproduct.

Lecture4>Section1>GROUPBY

Purchase(product, date, price, quantity)

12

GroupingandAggregation

1.ComputetheFROM andWHERE clauses

2.GroupbytheattributesintheGROUPBY

3.ComputetheSELECT clause:groupedattributesandaggregates

Semanticsofthequery:


13

1.ComputetheFROM andWHERE clauses

Product Date Price QuantityBagel 10/21 1 20Bagel 10/25 1.50 20Banana 10/3 0.5 10Banana 10/10 1 10

SELECT product, SUM(price*quantity) AS TotalSalesFROM PurchaseWHERE date > ‘10/1/2005’GROUP BY product


FROM

Product Date Price QuantityBagel 10/21 1 20Bagel 10/25 1.50 20Banana 10/3 0.5 10Banana 10/10 1 10

14

2.GroupbytheattributesintheGROUPBY

SELECT product, SUM(price*quantity) AS TotalSalesFROM PurchaseWHERE date > ‘10/1/2005’GROUP BY product

Lecture4 >Section1>GROUPBY

GROUP BY Product Date Price Quantity

Bagel10/21 1 2010/25 1.50 20

Banana10/3 0.5 1010/10 1 10

15

3.ComputetheSELECT clause:groupedattributesandaggregatesSELECT product, SUM(price*quantity) AS TotalSalesFROM PurchaseWHERE date > ‘10/1/2005’GROUP BY product

Product TotalSales

Bagel 50

Banana 15

SELECTProduct Date Price Quantity

Bagel10/21 1 2010/25 1.50 20

Banana10/3 0.5 1010/10 1 10


16

GROUPBYv.s.NestedQuereis

SELECT product, Sum(price*quantity) AS TotalSalesFROM PurchaseWHERE date > ‘10/1/2005’GROUP BY product

SELECT DISTINCT x.product, (SELECT Sum(y.price*y.quantity)FROM Purchase yWHERE x.product = y.product

AND y.date > ‘10/1/2005’) AS TotalSalesFROM Purchase xWHERE x.date > ‘10/1/2005’


17

HAVINGClause

Samequeryasbefore,exceptthatweconsideronlyproductsthathavemorethan100buyers

HAVINGclausescontainsconditionsonaggregates

SELECT product, SUM(price*quantity)FROM PurchaseWHERE date > ‘10/1/2005’GROUP BY productHAVING SUM(quantity) > 100

WhereasWHEREclausesconditiononindividualtuples…


18

GeneralformofGroupingandAggregation

• S=CanONLYcontainattributesa1,…,ak and/oraggregatesoverotherattributes• C1 =isanyconditionontheattributesinR1,…,Rn• C2 =isanyconditionontheaggregateexpressions

SELECT SFROM R1,…,RnWHERE C1GROUP BY a1,…,akHAVING C2

Why?


19

GeneralformofGroupingandAggregationSELECT SFROM R1,…,RnWHERE C1GROUP BY a1,…,akHAVING C2

Evaluationsteps:1. EvaluateFROM-WHERE:applyconditionC1 onthe

attributesinR1,…,Rn2. GROUPBYtheattributesa1,…,ak3. ApplyconditionC2 toeachgroup(mayhaveaggregates)4. ComputeaggregatesinSandreturntheresult


20

Group-byv.s.NestedQuery

• Findauthorswhowrote³ 10documents:• Attempt1:withnestedqueries

SELECT DISTINCT Author.nameFROM AuthorWHERE COUNT(

SELECT Wrote.urlFROM WroteWHERE Author.login = Wrote.login) > 10

Author(login, name)Wrote(login, url)

ThisisSQLbyanovice


21

Group-byv.s.NestedQuery

• Findallauthorswhowroteatleast10documents:• Attempt2:SQLstyle(withGROUPBY)

SELECT Author.nameFROM Author, WroteWHERE Author.login = Wrote.loginGROUP BY Author.nameHAVING COUNT(Wrote.url) > 10

NoneedforDISTINCT:automaticallyfromGROUPBY

ThisisSQLbyanexpert


Group-byvs.NestedQuery

Whichwayismoreefficient?

• Attempt#1-Withnested:HowmanytimesdowedoaSFWqueryoveralloftheWroterelations?

• Attempt#2-Withgroup-by:Howaboutwhenwrittenthisway?

WithGROUPBYcanbemuch moreefficient!


Activity-4-1.ipynb

23

Lecture4>Section1 >ACTIVITY

3.AdvancedSQL-izing

24

Lecture4>Section2

Whatyouwilllearnaboutinthissection

1. Quantifiers

2. NULLs

3. OuterJoins

4. ACTIVITY:FancySQLPt.II

25

Lecture4>Section2

26

QuantifiersProduct(name, price, company)Company(name, city)

Findallcompaniesthatmakesomeproductswithprice<100

SELECT DISTINCT Company.cnameFROM Company, ProductWHERE Company.name = Product.company

AND Product.price < 100

Existential:easy!J

Lecture4>Section2 >Quantifiers

Anexistentialquantifier isalogicalquantifier(roughly)oftheform“thereexists”

27

QuantifiersProduct(name, price, company)Company(name, city)

Findallcompanieswithproductsallhavingprice<100

SELECT DISTINCT Company.cnameFROM CompanyWHERE Company.name NOT IN(

SELECT Product.companyFROM Product.price >= 100)

Auniversalquantifier isoftheform“forall” Universal:hard!L

Findallcompaniesthatmakeonlyproductswithprice<100

Equivalent

Lecture4>Section2 >Quantifiers

28

NULLSinSQL• Wheneverwedon’thaveavalue,wecanputaNULL

• Canmeanmanythings:• Valuedoesnotexists• Valueexistsbutisunknown• Valuenotapplicable• Etc.

• Theschemaspecifiesforeachattributeifcanbenull(nullable attribute)ornot

• HowdoesSQLcopewithtablesthathaveNULLs?

Lecture4 >Section2>NULLs

29

NullValues

• Fornumericaloperations,NULL->NULL:• Ifx=NULLthen4*(3-x)/7isstillNULL

• Forboolean operations,inSQLtherearethreevalues:

FALSE= 0UNKNOWN= 0.5TRUE= 1

• Ifx=NULLthenx=“Joe”isUNKNOWN


30

NullValues

• C1ANDC2=min(C1,C2)• C1OR C2=max(C1,C2)• NOTC1=1– C1

SELECT *FROM PersonWHERE (age < 25)

AND (height > 6 AND weight > 190)

Won’treturne.g.(age=20height=NULLweight=200)!

RuleinSQL:includeonlytuplesthatyieldTRUE(1.0)


31

NullValues

Unexpectedbehavior:

SELECT *FROM PersonWHERE age < 25 OR age >= 25

SomePersonsarenotincluded!


32

NullValues

CantestforNULLexplicitly:• xISNULL• xISNOTNULL

SELECT *FROM PersonWHERE age < 25 OR age >= 25

OR age IS NULL

NowitincludesallPersons!


33

RECAP:InnerJoinsBy default,joinsinSQLare“innerjoins”:

SELECT Product.name, Purchase.storeFROM Product JOIN Purchase ON Product.name = Purchase.prodName

SELECT Product.name, Purchase.storeFROM Product, PurchaseWHERE Product.name = Purchase.prodName

Product(name, category)Purchase(prodName, store)

Bothequivalent:BothINNERJOINS!


34

InnerJoins+NULLS=Lostdata?By default,joinsinSQLare“innerjoins”:

However:Productsthatneversold(withnoPurchasetuple)willbelost!

SELECT Product.name, Purchase.storeFROM Product JOIN Purchase ON Product.name = Purchase.prodName

SELECT Product.name, Purchase.storeFROM Product, PurchaseWHERE Product.name = Purchase.prodName

Product(name, category)Purchase(prodName, store)


35

OuterJoins

• Anouterjoin returnstuplesfromthejoinedrelationsthatdon’thaveacorrespondingtupleintheotherrelations• I.e.IfwejoinrelationsAandBona.X =b.X,andthereisanentryinAwithX=5,butnoneinBwithX=5…• ALEFTOUTERJOINwillreturnatuple(a,NULL)!

• LeftouterjoinsinSQL:

Lecture4>Section2>OuterJoins

SELECT Product.name, Purchase.storeFROM Product LEFT OUTER JOIN Purchase ON

Product.name = Purchase.prodName

Nowwe’llgetproductseveniftheydidn’tsell

36

name category

Gizmo gadget

Camera Photo

OneClick Photo

prodName store

Gizmo Wiz

Camera Ritz

Camera Wiz

name store

Gizmo Wiz

Camera Ritz

Camera Wiz

Product PurchaseINNERJOIN:

SELECT Product.name, Purchase.storeFROM Product

INNER JOIN Purchase ON Product.name = Purchase.prodName

Note:anotherequivalentwaytowriteanINNERJOIN!


37

name category

Gizmo gadget

Camera Photo

OneClick Photo

prodName store

Gizmo Wiz

Camera Ritz

Camera Wiz

name store

Gizmo Wiz

Camera Ritz

Camera Wiz

OneClick NULL

Product PurchaseLEFTOUTERJOIN:

SELECT Product.name, Purchase.storeFROM Product

LEFT OUTER JOIN Purchase ON Product.name = Purchase.prodName


38

OtherOuterJoins

• Leftouterjoin:• Includethelefttupleevenifthere’snomatch

• Rightouterjoin:• Includetherighttupleevenifthere’snomatch

• Fullouterjoin:• Includethebothleftandrighttuplesevenifthere’snomatch


Activity-4-2.ipynb

39

Lecture3>Section3>ACTIVITY

Summary

SQLisarichprogramminglanguagethathandlesthewaydataisprocessed

declaratively

40

Lecture2,3,&4>SUMMARY

ProblemSet#1:SQLUberAlles

Lecture4>ProblemSet#1

ProblemsinPS#1

42


1. LinearalgebrainSQL

2. Precipitationdataandnestedqueries

3. ThetravelingSQLsalesman:GraphtraversalsinSQL

LinearalgebrainSQL

1. Simplejoinswithaggregations

2. Hint1:UsingaliasesleadstocleanSQL

3. Hint2:SQLsupportsmanyoperationsovernumericattributes(intheSELECTpartofanSFWquery)

43


SELECT MAX(A.val*B.val)FROM A, BWHERE A.i= B.i AND A.j = B.j

Precipitationdataandnestedqueries

1. Aggregatesinsidenestedqueries.RememberSQLiscompositional

2. Hint1:Breakdownquerydescriptiontosteps(subproblems)

3. Hint2:Wheneverindoubtalwaysgobacktothedefinition

44


PrecipitationdataandnestedqueriesExample:“Usinga singleSQLquery,findallofthestationsthathadthehighestdailyprecipitation(acrossallstations)onanygivenday.”

45


SELECT station_id, dayFROM precipitation,

(SELECT day AS maxd, MAX(precipitation)AS maxpFROM precipitationGROUP BY day)

WHERE day = maxd AND precipitation = maxp

Precipitation

ThetravelingSQLsalesman:GraphtraversalsinSQL

1. Views:Detailsinthedescription.Nothingmorethantempaliasesforqueries.Remember:SQLiscompositional!

2. Self-joinsareverypowerful

46


Example:“Findallpathsofsizetwoinadirectedgraph”

47


SELECT e1.src, e1.trg, e2.trgFROM edges AS e1, edges AS e2, WHERE e1.trg = e2.src

ThetravelingSQLsalesman:GraphtraversalsinSQL

edge_id src trg

1 A B

2 B C

3 C D

Edges

Somemoreexamples:https://www.fusionbox.com/blog/detail/graph-algorithms-in-a-database-recursive-ctes-and-topological-sort-with-postgres/620/

Lecture 4 SQL Adv II - GitHub Pages · •I.e. If we join relations A and B on a.X= b.X, and there...

Documents

Transcript of Lecture 4 SQL Adv II - GitHub Pages · •I.e. If we join relations A and B on a.X= b.X, and there...