Post on 29-Jul-2015
© 2015 by Markus WinandiStockPhoto/Mitshu
From Backend to FrontendWhen database optimization affects the full stack
@MarkusWinand@SQLPerfTips@ModernSQL
© 2015 by Markus Winand
Full Stack, Really?
© 2015 by Markus Winand
Full Stack, Really?
© 2015 by Markus Winand
What Users Want are Used to
© 2015 by Markus Winand
What Users Want are Used to
Let’sclick here
© 2015 by Markus Winand
User Clicks on Page 2
GimmePage
2!
Gimme Rows11 to
20
SQL for “Gimme Rows 11 to 20”
se!e"# * $%om news whe%e #op&" ' 1234 o%de% b( da#e des", &d des" o!!se" 10 #$m$" 10
Inside the Database: Worst Case
L&m&# (a"#ua! %ows'10)-> So%# (a%"ua# &ows'20) So%# Me#hod: "op-N heapso&" Memo&(: 19)B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)
Inside the Database: Worst Case
L&m&# (a"#ua! %ows'10)-> So%# (a%"ua# &ows'30) So%# Me#hod: "op-N heapso&" Memo&(: 20)B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)
Inside the Database: Worst Case
L&m&# (a"#ua! %ows'10)-> So%# (a%"ua# &ows'40) So%# Me#hod: "op-N heapso&" Memo&(: 22)B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)
Inside the Database: Worst Case
L&m&# (a"#ua! %ows'10)-> So%# (a%"ua# &ows'10000) So%# Me#hod: ex"e&na# me&*e D$s): 1200)B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)
Inside the Database: Worst Case
Speed depends on the number of matched rows and the page fetched.
Fetching the last pagecan take considerablylonger than fetchingthe first page.
Improvement 1: Indexed order by
se!e"# * $%om news whe%e #op&" ' 1234 o%de% b( da#e des", &d des" o$$se# 10 !&m&# 10;
%&ea"e $ndex on news ("op$%, da"e, $d)
A single index to support the whe%e and o%de% b( clauses.
Improvement 1: Indexed order by
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'10) Index *ond: (#op&" ' 0)
Improvement 1: Indexed order by
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'20) Index *ond: (#op&" ' 0)
Improvement 1: Indexed order by
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'30) Index *ond: (#op&" ' 0)
Improvement 1: Indexed order by
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'40) Index *ond: (#op&" ' 0)
Improvement 1: Indexed order by
L&m&# (a"#ua! %ows'10)-> So%# (a%"ua# &ows'10000) So%# Me#hod: ex"e&na# me&*e D$s): 1200)B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)
Improvement 1: Indexed order by
Fetching the first page is not affected by theBase-Set size!
Fetching the next pageis also faster. However, the database might revert to the other execution plan when browsing to the end.
© 2013 by Markus Winand
It’s not just about performance
© 2015 by Markus Winand
Offsets Yield Unstable Results
© 2015 by Markus Winand
Offsets Yield Unstable Results
© 2015 by Markus Winand
Offsets Yield Unstable Results
© 2013 by Markus Winand
Why don’t we ask for the next rows after...
© 2015 by Markus Winand
Asking for “Next After” in SQL
© 2015 by Markus Winand
Asking for “Next After” in SQL
© 2015 by Markus Winand
Asking for “Next After” in SQL
© 2015 by Markus Winand
Asking for “Next After” in SQL
© 2015 by Markus Winand
Asking for “Next After” in SQL
© 2015 by Markus Winand
Asking for “Next After” in SQL
SQL For “Next After” (Key-Set Pagination)
se!e"# * $%om news whe%e #op&" ' 1234 and (da"e, $d) < (?, ?) o%de% b( da#e des", &d des" !&m&# 10 Look Ma,
no offset!
Key-Set Pagination: Worst Case
Key-Set Pagination: Worst Case
L&m&# (a%"ua# &ows'10)-> So%# (a%"ua# &ows'10) So%# Me#hod: "op-N heapso&" Memo&(: 18)B -> B&#map Heap S"an (&ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (&ows'10000) Index *ond: (#op&" ' 1234)
Key-Set Pagination: Worst Case
L&m&# (a"#ua! %ows'10) -> So%# (a%"ua# &ows'10) So%# Me#hod: "op-N heapso&" Memo&(: 18)B -> B&#map Heap S"an (a%"ua# &ows'9990) Rows Remo+ed b( F$#"e&: 10 (new &n 9.2) -> B&#map Index S"an (a%"ua# &ows'10000) Index *ond: (#op&" ' 1234)
Key-Set Pagination: Worst Case
L&m&# (a"#ua! %ows'10) -> So%# (a%"ua# &ows'10) So%# Me#hod: "op-N heapso&" Memo&(: 18)B -> B&#map Heap S"an (a%"ua# &ows'9980) Rows Remo+ed b( F$#"e&: 20 (new &n 9.2) -> B&#map Index S"an (a%"ua# &ows'10000) Index *ond: (#op&" ' 1234)
Key-Set Pagination: Worst Case
L&m&# (a"#ua! %ows'10) -> So%# (a%"ua# &ows'10) So%# Me#hod: "op-N heapso&" Memo&(: 18)B -> B&#map Heap S"an (a%"ua# &ows'9970) Rows Remo+ed b( F$#"e&: 30 (new &n 9.2) -> B&#map Index S"an (a%"ua# &ows'10000) Index *ond: (#op&" ' 1234)
Key-Set Pagination: Worst Case
L&m&# (a"#ua! %ows'10) -> So%# (a%"ua# &ows'10) So%# Me#hod: "op-N heapso&" Memo&(: 18)B -> B&#map Heap S"an (a%"ua# &ows'10) Rows Remo+ed b( F$#"e&: 9990 (new &n 9.2) -> B&#map Index S"an (a%"ua# &ows'10000) Index *ond: (#op&" ' 1234)
Key-Set Pagination w/o Index for order by
Always needs to retrieve the full base set, but the top-n sort buffer needs to hold only 10 rows.
The response time remains the same even whenbrowsing to the last page.And the memory footprintis very low!
Key-Set Pagination: Best Case
se!e"# * $%om news whe%e #op&" ' 1234 and (da#e, &d) < (?, ?) o%de% b( da#e des", &d des" !&m&# 10;
%&ea"e $ndex on news ("op$%, da"e, $d)
Key-Set Pagination: Best Case
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'10) Index *ond: (#op&" ' 1234)
Key-Set Pagination: Best Case
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'10) Index *ond: ((#op&" ' 1234) AND (ROW(d#, &d) < ROW(‘...’, 23456)))
Key-Set Pagination: Best Case
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'10) Index *ond: ((#op&" ' 1234) AND (ROW(d#, &d) < ROW(‘...’, 34567)))
Key-Set Pagination: Best Case
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'10) Index *ond: ((#op&" ' 1234) AND (ROW(d#, &d) < ROW(‘...’, 45678)))
Key-Set Pagination: Best Case
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'10) Index *ond: ((#op&" ' 1234) AND (ROW(d#, &d) < ROW(‘...’, 56789)))
Key-Set Pagination: Best Case
Successively browsing back doesn’t slow down.
Neither the size of thebase set(*) nor the fetched page number affects the response time.
(*) the index tree depth still affects the response time.
ComparisonO
ffset
Key
-Set
Worst Case Best Case
© 2015 by Markus Winand
Too good to be true?
Key-Set Pagination has serious limitations:
‣You cannot directly navigate to arbitrary pages‣because you need the values from the previous page
‣Bi-directional navigation is possible but tedious‣you need to reverse the o%de% b( direction and RV comparison
‣Tooling...???
© 2015 by Markus Winand
Required Tooling for Key-Set Pagination
© 2015 by Markus Winand
Required Tooling for Key-Set Pagination
Proper Row-Values Support.
Don’t use OFFSET.NoSQL is
also Also for NoSQL
© 2015 by Markus Winand
Required Tooling for Key-Set Pagination
Proper Row-Values Support.
Don’t use OFFSET.
Offer Infinite Scrolling.
Don’t think in pages.
Ask the right question.
OFFSET is EVIL
http://use-the-index-luke.com/no-offset
© 2015 by Markus Winand
About Markus Winand
Tuning developers forhigh SQL performance
Training & co (one-man show): winand.at
Geeky blog: use-the-index-luke.com
Author of: SQL Performance Explained