2014 10-23-oopsla-i3ql
-
Upload
sebastian-erdweg -
Category
Software
-
view
310 -
download
0
description
Transcript of 2014 10-23-oopsla-i3ql
i3QL: Language-Integrated Live Data Views
Ralf Mitschke Sebastian Erdweg Mirko Köhler Mira Mezini Guido Salvaneschi
�������
��� ������� �
����
��������������
��
����� �
����������
�������
�������� �
���
2
World airportsFlight data
2
World airportsFlight data
From each city, how many flights to Portland in 2014?
3
World airports
Flight data
FROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ !
!
!
From each city, how many flights to Portland in 2014?
3
World airports
Flight data
FROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ !
!
!
AND f.takeoff >= '2014-01-01' AND f.takeoff <= '2014-12-31'
From each city, how many flights to Portland in 2014?
3
World airports
Flight data
FROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ !
!
!
AND f.takeoff >= '2014-01-01' AND f.takeoff <= '2014-12-31'
GROUP BY a1.city
From each city, how many flights to Portland in 2014?
3
World airports
Flight data
FROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ !
!
!
SELECT a1.city AS From COUNT(*) AS Flights
AND f.takeoff >= '2014-01-01' AND f.takeoff <= '2014-12-31'
GROUP BY a1.city
From each city, how many flights to Portland in 2014?
4
World airports
Flight data
SELECT a1.city AS From COUNT(*) AS FlightsFROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ AND f.takeoff >= '2014-01-01' AND f.takeoff <= '2014-12-31'GROUP BY a1.city
From each city, how many flights to Portland in 2014?
5
World airports
Flight data
SELECT a1.city AS From COUNT(*) AS FlightsFROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ AND f.takeoff >= '2014-01-01' AND f.takeoff <= '2014-12-31'GROUP BY a1.city
ID Code City1 AMS Amsterdam2 BOS Boston3 SFO San Francisco 4 NRT Tokyo5 PDX Portland
5
World airports
Flight data
SELECT a1.city AS From COUNT(*) AS FlightsFROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ AND f.takeoff >= '2014-01-01' AND f.takeoff <= '2014-12-31'GROUP BY a1.city
ID Code City1 AMS Amsterdam2 BOS Boston3 SFO San Francisco 4 NRT Tokyo5 PDX Portland
From To Take off1 5 2014-01-01 10:00am1 5 2014-01-03 10:00am1 2 2014-01-01 1:20pm4 5 2013-12-31 4:00pm4 5 2014-01-01 4:00pm2 5 2014-12-31 5:05pm
5
World airports
Flight data
From Flights
Amsterdam 360
Boston 350
San Francisco 3467
Tokyo 349
SELECT a1.city AS From COUNT(*) AS FlightsFROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ AND f.takeoff >= '2014-01-01' AND f.takeoff <= '2014-12-31'GROUP BY a1.city
ID Code City1 AMS Amsterdam2 BOS Boston3 SFO San Francisco 4 NRT Tokyo5 PDX Portland
From To Take off1 5 2014-01-01 10:00am1 5 2014-01-03 10:00am1 2 2014-01-01 1:20pm4 5 2013-12-31 4:00pm4 5 2014-01-01 4:00pm2 5 2014-12-31 5:05pm
6
World airports
Flight data
From Flights
Amsterdam 360
Boston 350
San Francisco 3467
Tokyo 349
SELECT a1.city AS From COUNT(*) AS FlightsFROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ AND f.takeoff >= '2014-01-01' AND f.takeoff <= '2014-12-31'GROUP BY a1.city
ID Code City1 AMS Amsterdam2 BOS Boston3 SFO San Francisco 4 NRT Tokyo5 PDX Portland
From To Take off1 5 2014-01-01 10:00am1 5 2014-01-03 10:00am1 2 2014-01-01 1:20pm4 5 2013-12-31 4:00pm4 5 2014-01-01 4:00pm2 5 2014-12-31 5:05pm
6
World airports
Flight data
From Flights
Amsterdam 360
Boston 350
San Francisco 3467
Tokyo 349
SELECT a1.city AS From COUNT(*) AS FlightsFROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ AND f.takeoff >= '2014-01-01' AND f.takeoff <= '2014-12-31'GROUP BY a1.city
ID Code City1 AMS Amsterdam2 BOS Boston3 SFO San Francisco 4 NRT Tokyo5 PDX Portland
From To Take off1 5 2014-01-01 10:00am1 5 2014-01-03 10:00am1 2 2014-01-01 1:20pm4 5 2013-12-31 4:00pm4 5 2014-01-01 4:00pm2 5 2014-12-31 5:05pm3 5 2014-09-14 8:15pm Additional flight
6
World airports
Flight data
From Flights
Amsterdam 360
Boston 350
San Francisco 3467
Tokyo 349
SELECT a1.city AS From COUNT(*) AS FlightsFROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ AND f.takeoff >= '2014-01-01' AND f.takeoff <= '2014-12-31'GROUP BY a1.city
ID Code City1 AMS Amsterdam2 BOS Boston3 SFO San Francisco 4 NRT Tokyo5 PDX Portland
From To Take off1 5 2014-01-01 10:00am1 5 2014-01-03 10:00am1 2 2014-01-01 1:20pm4 5 2013-12-31 4:00pm4 5 2014-01-01 4:00pm2 5 2014-12-31 5:05pm3 5 2014-09-14 8:15pm Additional flight
+1
From To Take off1 5 2014-01-01 10:00am1 5 2014-01-03 10:00am1 2 2014-01-01 1:20pm4 5 2013-12-31 4:00pm4 5 2014-01-01 4:00pm2 5 2014-12-31 5:05pm3 5 2014-09-14 8:15pm
7
World airports
Flight data
From Flights
Amsterdam 360
Boston 350
San Francisco 3468
Tokyo 349
SELECT a1.city AS From COUNT(*) AS FlightsFROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ AND f.takeoff >= '2014-01-01' AND f.takeoff <= '2014-12-31'GROUP BY a1.city
ID Code City1 AMS Amsterdam2 BOS Boston3 SFO San Francisco 4 NRT Tokyo5 PDX Portland
From To Take off1 5 2014-01-01 10:00am1 5 2014-01-03 10:00am1 2 2014-01-01 1:20pm4 5 2013-12-31 4:00pm4 5 2014-01-01 4:00pm2 5 2014-12-31 5:05pm3 5 2014-09-14 8:15pm2 5 2015-01-01 11:05am
7
World airports
Flight data
From Flights
Amsterdam 360
Boston 350
San Francisco 3468
Tokyo 349
SELECT a1.city AS From COUNT(*) AS FlightsFROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ AND f.takeoff >= '2014-01-01' AND f.takeoff <= '2014-12-31'GROUP BY a1.city
ID Code City1 AMS Amsterdam2 BOS Boston3 SFO San Francisco 4 NRT Tokyo5 PDX Portland
Rescheduled
From To Take off1 5 2014-01-01 10:00am1 5 2014-01-03 10:00am1 2 2014-01-01 1:20pm4 5 2013-12-31 4:00pm4 5 2014-01-01 4:00pm2 5 2014-12-31 5:05pm3 5 2014-09-14 8:15pm2 5 2015-01-01 11:05am
7
World airports
Flight data
From Flights
Amsterdam 360
Boston 350
San Francisco 3468
Tokyo 349
SELECT a1.city AS From COUNT(*) AS FlightsFROM Airport a1, Airport a2, Flight fWHERE f.from = a1.id AND f.to = a2.id AND a2.code = ‘PDX’ AND f.takeoff >= '2014-01-01' AND f.takeoff <= '2014-12-31'GROUP BY a1.city
ID Code City1 AMS Amsterdam2 BOS Boston3 SFO San Francisco 4 NRT Tokyo5 PDX Portland
-1
Rescheduled
8
8
8
ChangeUpdate
8
ChangeUpdate
9
Maintain live data views
9
Maintain live data views
= Incremental computations
10
Incremental computations
at our fingertips
10
Incremental computations
at our fingertips(inside PL)
11
i3QL!integrated! ! ! ! !in-memory! ! ! ! !incremental! ! !
11
i3QL!integrated! ! ! ! !in-memory! ! ! ! !incremental! ! !
Scala-embedded DSL
11
i3QL!integrated! ! ! ! !in-memory! ! ! ! !incremental! ! !
Scala-embedded DSLcollections API
11
i3QL!integrated! ! ! ! !in-memory! ! ! ! !incremental! ! !
Scala-embedded DSLcollections APIbased on relational algebra
11
i3QL!integrated! ! ! ! !in-memory! ! ! ! !incremental! ! !
Scala-embedded DSLcollections APIbased on relational algebra
case class Airport(id: Int, code: String, city: String)case class Flight(from: Int, to: Int, takeoff: Date)
11
i3QL!integrated! ! ! ! !in-memory! ! ! ! !incremental! ! !
Scala-embedded DSLcollections APIbased on relational algebra
case class Airport(id: Int, code: String, city: String)case class Flight(from: Int, to: Int, takeoff: Date)
val airport: Table[Airport]val flight: Table[Flight]
11
i3QL!integrated! ! ! ! !in-memory! ! ! ! !incremental! ! !
Scala-embedded DSLcollections APIbased on relational algebra
case class Airport(id: Int, code: String, city: String)case class Flight(from: Int, to: Int, takeoff: Date)
val airport: Table[Airport]val flight: Table[Flight]
airport += Airport(1, "AMS", "Amsterdam")airport += Airport(2, "BOS", "Boston")airport += Airport(5, "PDX", “Portland")
11
i3QL!integrated! ! ! ! !in-memory! ! ! ! !incremental! ! !
Scala-embedded DSLcollections APIbased on relational algebra
case class Airport(id: Int, code: String, city: String)case class Flight(from: Int, to: Int, takeoff: Date)
val airport: Table[Airport]val flight: Table[Flight]
airport += Airport(1, "AMS", "Amsterdam")airport += Airport(2, "BOS", "Boston")airport += Airport(5, "PDX", “Portland")
flight += Flight(1, 5, new Date(2014, 1, 1, 10, 0))flight += Flight(1, 5, new Date(2014, 1, 3, 10, 0))flight += Flight(2, 5, new Date(2014, 12, 31, 17, 5))
12
i3QL queriesval airport: Table[Airport]val flight: Table[Flight]!
val q = ( !
!
!
!
!
!
!
!
!
!
!
)
FROM (airport, airport, flight)WHERE ((a1, a2, f) => f.from == a1.id AND f.to == a2.id AND a2.code == "PDX" !
)
12
i3QL queriesval airport: Table[Airport]val flight: Table[Flight]!
val q = ( !
!
!
!
!
!
!
!
!
!
!
)
FROM (airport, airport, flight)WHERE ((a1, a2, f) => f.from == a1.id AND f.to == a2.id AND a2.code == "PDX" !
)
a2.code == "PDX" AND f.takeoff >= new Date(2014, 01, 01) AND f.takeoff < new Date(2015, 01, 01)
12
i3QL queriesval airport: Table[Airport]val flight: Table[Flight]!
val q = ( !
!
!
!
!
!
!
!
!
!
!
)
FROM (airport, airport, flight)WHERE ((a1, a2, f) => f.from == a1.id AND f.to == a2.id AND a2.code == "PDX" !
)
a2.code == "PDX" AND f.takeoff >= new Date(2014, 01, 01) AND f.takeoff < new Date(2015, 01, 01) GROUP BY ((a1: Rep[Airport], a2: Rep[Airport], f: Rep[Flight]) => a1.city)
12
i3QL queriesval airport: Table[Airport]val flight: Table[Flight]!
val q = ( !
!
!
!
!
!
!
!
!
!
!
)
FROM (airport, airport, flight)WHERE ((a1, a2, f) => f.from == a1.id AND f.to == a2.id AND a2.code == "PDX" !
)
a2.code == "PDX" AND f.takeoff >= new Date(2014, 01, 01) AND f.takeoff < new Date(2015, 01, 01) GROUP BY ((a1: Rep[Airport], a2: Rep[Airport], f: Rep[Flight]) => a1.city)
SELECT ((city: Rep[String]) => city, COUNT(*))
13
i3QL queriesval airport: Table[Airport] val flight: Table[Flight] val q = SELECT … FROM … WHERE …
13
i3QL queriesval airport: Table[Airport] val flight: Table[Flight] val q = SELECT … FROM … WHERE …
val result = q.asMaterialized
13
i3QL queriesval airport: Table[Airport] val flight: Table[Flight] val q = SELECT … FROM … WHERE …
val result = q.asMaterialized
From Flights
Amsterdam 360
Boston 350
San Francisco 3467
Tokyo 349
14
i3QL queriesval airport: Table[Airport] val flight: Table[Flight] !
val q = SELECT … FROM … WHERE … !
val result = q.asMaterialized !
flight += Flight(3, 5, new Date(2014, 9, 14, 20, 15))
From Flights
Amsterdam 360
Boston 350
San Francisco 3467
Tokyo 349
14
i3QL queriesval airport: Table[Airport] val flight: Table[Flight] !
val q = SELECT … FROM … WHERE … !
val result = q.asMaterialized !
flight += Flight(3, 5, new Date(2014, 9, 14, 20, 15))
From Flights
Amsterdam 360
Boston 350
San Francisco 3467
Tokyo 349
+1
15
i3QL queriesval airport: Table[Airport] val flight: Table[Flight] !
val q = SELECT … FROM … WHERE … !
val result = q.asMaterialized !
flight += Flight(3, 5, new Date(2014, 9, 14, 20, 15))flight ~= Flight(2, 5, new Date(2014, 12, 31, 17, 5)) -> Flight(2, 5, new Date(2015, 1, 1, 11, 5))
From Flights
Amsterdam 360
Boston 350
San Francisco 3468
Tokyo 349
15
i3QL queriesval airport: Table[Airport] val flight: Table[Flight] !
val q = SELECT … FROM … WHERE … !
val result = q.asMaterialized !
flight += Flight(3, 5, new Date(2014, 9, 14, 20, 15))flight ~= Flight(2, 5, new Date(2014, 12, 31, 17, 5)) -> Flight(2, 5, new Date(2015, 1, 1, 11, 5))
From Flights
Amsterdam 360
Boston 350
San Francisco 3468
Tokyo 349
-1
16
i3QL query execution
16
i3QL query execution
1.Translate i3QL query to relational-algebra tree
16
i3QL query execution
1.Translate i3QL query to relational-algebra tree
2.Use incremental relational-algebra operatorsthat propagate change events
17
×
airport flight
17
×
airport flight
σ f.from == a1.idf.to == a2.ida2.code == “PDX”f.takeoff >= new Date(2014, 01, 01)f.takeoff < new Date(2015, 01, 01)
17
×
airport flight
σ f.from == a1.idf.to == a2.ida2.code == “PDX”f.takeoff >= new Date(2014, 01, 01)f.takeoff < new Date(2015, 01, 01)
γ GROUP a1.cityCOUNT(*)
17
×
airport flight
σ f.from == a1.idf.to == a2.ida2.code == “PDX”f.takeoff >= new Date(2014, 01, 01)f.takeoff < new Date(2015, 01, 01)
γ GROUP a1.cityCOUNT(*)
add((3, 5, new Date(2014, 9, 14, 20, 15)))
17
×
airport flight
σ f.from == a1.idf.to == a2.ida2.code == “PDX”f.takeoff >= new Date(2014, 01, 01)f.takeoff < new Date(2015, 01, 01)
γ GROUP a1.cityCOUNT(*)
add((3, 5, new Date(2014, 9, 14, 20, 15)))
for (a1, a2 <- airport) add(a1, a2, (3, 5, new Date(2014, 9, 14, 20, 15)))
17
×
airport flight
σ f.from == a1.idf.to == a2.ida2.code == “PDX”f.takeoff >= new Date(2014, 01, 01)f.takeoff < new Date(2015, 01, 01)
γ GROUP a1.cityCOUNT(*)
add((3, 5, new Date(2014, 9, 14, 20, 15)))
for (a1, a2 <- airport) add(a1, a2, (3, 5, new Date(2014, 9, 14, 20, 15)))
add((3, “SFO”), (5, “PDX”), (3, 5, new Date(2014, 9, 14, 20, 15)))
17
×
airport flight
σ f.from == a1.idf.to == a2.ida2.code == “PDX”f.takeoff >= new Date(2014, 01, 01)f.takeoff < new Date(2015, 01, 01)
γ GROUP a1.cityCOUNT(*)
add((3, 5, new Date(2014, 9, 14, 20, 15)))
for (a1, a2 <- airport) add(a1, a2, (3, 5, new Date(2014, 9, 14, 20, 15)))
add((3, “SFO”), (5, “PDX”), (3, 5, new Date(2014, 9, 14, 20, 15)))
update((“San Francisco”, 3467) -> (“San Francisco”, 3468))
18
i3QL query execution
1.Translate i3QL query to relational-algebra tree
!
3.Use incremental relational-algebra operatorsthat propagate change events
18
i3QL query execution
1.Translate i3QL query to relational-algebra tree
!
3.Use incremental relational-algebra operatorsthat propagate change events
} based on LMSby Rompf et al.
1.
2.Optimize relational-algebra tree !! - algebraic laws!! - automatic indexing !! - partial evaluation !! - common subquery elimination
19
×
airport flight
σf.from == a1.idf.to == a2.id
a2.code == “PDX”f.takeoff >= new Date(2014, 01, 01)f.takeoff < new Date(2015, 01, 01)
γGROUP a1.cityCOUNT(*)
optimize
19
×
airport flight
σf.from == a1.idf.to == a2.id
a2.code == “PDX”f.takeoff >= new Date(2014, 01, 01)f.takeoff < new Date(2015, 01, 01)
γGROUP a1.cityCOUNT(*)
airport flight
optimize
σcode==“PDX” σ takeoff >= new Date(2014, 01, 01)takeoff < new Date(2015, 01, 01)
19
×
airport flight
σf.from == a1.idf.to == a2.id
a2.code == “PDX”f.takeoff >= new Date(2014, 01, 01)f.takeoff < new Date(2015, 01, 01)
γGROUP a1.cityCOUNT(*)
airport flight
optimize
σcode==“PDX” σ takeoff >= new Date(2014, 01, 01)takeoff < new Date(2015, 01, 01)
⋈ f.from == a1.idf.to == a2.id
automatic indexing
19
×
airport flight
σf.from == a1.idf.to == a2.id
a2.code == “PDX”f.takeoff >= new Date(2014, 01, 01)f.takeoff < new Date(2015, 01, 01)
γGROUP a1.cityCOUNT(*)
airport flight
optimize
σcode==“PDX”
γ GROUP a1.cityCOUNT(*)
σ takeoff >= new Date(2014, 01, 01)takeoff < new Date(2015, 01, 01)
⋈ f.from == a1.idf.to == a2.id
automatic indexing
20
i3QL operatorsOperator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERENOT EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1UNION SELECT ... FROM t2
Union all SELECT ... FROM t1UNION ALL SELECT ... FROM t2
Intersection SELECT ... FROM t1INTERSECT SELECT ... FROM t2
Table 1: i3QL operators.
The basis of the change propagation is the Observer inter-face depicted in Figure 3. The interface declares three atomicchange events for addition, removal and update.
1 trait Observer[V] {2 def added(v: V)3 def removed(v: V)4 def updated(oldV: V, newV: V)5 }
Figure 3: The observer interface for change propagation.
Every i3QL operator implements the Observer trait and acorresponding Observable trait. The implementation of theObserver defines incremental semantics of an operator. Anoperator registers itself as an observer of its predecessors inthe graph and receives atomic change events through any ofthe three methods of Observer.
Selection Operator To illustrate the implementation of op-erators, we show the implementation of the selection oper-ator in Figure 4. The selection operator observes an under-lying relation containing elements of type V and filters theelements using a predicate function. Each operator has a type
for resulting elements, defined by the type argument passedto Observable. In case of selections,V is the input and outputtype of the operator. The change propagation of selection iscomparatively simple: Selection only propagates a changeto its observers if the involved value satisfies the predicate.Finally, selection registers itself as an observer of the under-lying relation.
1 class Selection[V](rel: Relation[V], predicate: V => Boolean)2 extends Observer[V] with Observable[V] {3 def added(v: V) { if(predicate(v)) notify added(v) }4 def removed(v: V) { if(predicate(v)) notify removed(v) }5 def updated(oldV: V, newV: V) {6 if(predicate(oldV)) {7 if(predicate(newV)) notify updated(oldV, newV)8 else notify removed(oldV) }9 else if(predicate(newV)) notify added(newV) }
10 rel addObserver this11 }
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator thatcorrelates elements of a left-hand and right-hand side rela-tion. The results are pairs of elements from the two rela-tions. Instead of a binary predicate expressing the join con-dition, our join operator uses two functions that map ele-ments to key values; the operator joins two elements if theyare mapped to the same key. For example, in the join exam-ple shown in Sec. 2, we compared the properties firstName
and lastName of elements from two relations. i3QL automat-ically translates this condition into two key functions of theform (s:Student) => (s.firstName, s.lastName), where the key isa tuple of all values that have to be equal.
1 class Join[V1, V2, Key](2 left: Relation[V1], right: Relation[V2],3 leftKeyFun: V1 => Key, rightKeyFun: V2 => Key)4 extends Observable[(V1,V2)] {5 val leftIndex : Map[Key, List[V1]] = ...6 val rightIndex : Map[Key, List[V2]] = ...7 object LeftObserver extends Observer[V1] {8 def added(v: V1) {9 val key = leftKeyFun(v)
10 rightIndex.get(key) match {11 case Some(list) =>12 list.foreach( u => notify added ((u,v)) )13 case => // no changes propagated14 }}...}15 object RightObserver extends Observer[V2] { ... }16 }
Figure 5: Excerpt of the incremental join operator.
Figure 5 shows an excerpt of the implementation of the in-cremental join operator. The join—and any binary operatorin general—observes the input relations via two differentlytyped observers (line 7 and 15). To correlate elements, a join
Operator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERENOT EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1UNION SELECT ... FROM t2
Union all SELECT ... FROM t1UNION ALL SELECT ... FROM t2
Intersection SELECT ... FROM t1INTERSECT SELECT ... FROM t2
Table 1: i3QL operators.
The basis of the change propagation is the Observer inter-face depicted in Figure 3. The interface declares three atomicchange events for addition, removal and update.
1 trait Observer[V] {2 def added(v: V)3 def removed(v: V)4 def updated(oldV: V, newV: V)5 }
Figure 3: The observer interface for change propagation.
Every i3QL operator implements the Observer trait and acorresponding Observable trait. The implementation of theObserver defines incremental semantics of an operator. Anoperator registers itself as an observer of its predecessors inthe graph and receives atomic change events through any ofthe three methods of Observer.
Selection Operator To illustrate the implementation of op-erators, we show the implementation of the selection oper-ator in Figure 4. The selection operator observes an under-lying relation containing elements of type V and filters theelements using a predicate function. Each operator has a type
for resulting elements, defined by the type argument passedto Observable. In case of selections,V is the input and outputtype of the operator. The change propagation of selection iscomparatively simple: Selection only propagates a changeto its observers if the involved value satisfies the predicate.Finally, selection registers itself as an observer of the under-lying relation.
1 class Selection[V](rel: Relation[V], predicate: V => Boolean)2 extends Observer[V] with Observable[V] {3 def added(v: V) { if(predicate(v)) notify added(v) }4 def removed(v: V) { if(predicate(v)) notify removed(v) }5 def updated(oldV: V, newV: V) {6 if(predicate(oldV)) {7 if(predicate(newV)) notify updated(oldV, newV)8 else notify removed(oldV) }9 else if(predicate(newV)) notify added(newV) }
10 rel addObserver this11 }
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator thatcorrelates elements of a left-hand and right-hand side rela-tion. The results are pairs of elements from the two rela-tions. Instead of a binary predicate expressing the join con-dition, our join operator uses two functions that map ele-ments to key values; the operator joins two elements if theyare mapped to the same key. For example, in the join exam-ple shown in Sec. 2, we compared the properties firstName
and lastName of elements from two relations. i3QL automat-ically translates this condition into two key functions of theform (s:Student) => (s.firstName, s.lastName), where the key isa tuple of all values that have to be equal.
1 class Join[V1, V2, Key](2 left: Relation[V1], right: Relation[V2],3 leftKeyFun: V1 => Key, rightKeyFun: V2 => Key)4 extends Observable[(V1,V2)] {5 val leftIndex : Map[Key, List[V1]] = ...6 val rightIndex : Map[Key, List[V2]] = ...7 object LeftObserver extends Observer[V1] {8 def added(v: V1) {9 val key = leftKeyFun(v)
10 rightIndex.get(key) match {11 case Some(list) =>12 list.foreach( u => notify added ((u,v)) )13 case => // no changes propagated14 }}...}15 object RightObserver extends Observer[V2] { ... }16 }
Figure 5: Excerpt of the incremental join operator.
Figure 5 shows an excerpt of the implementation of the in-cremental join operator. The join—and any binary operatorin general—observes the input relations via two differentlytyped observers (line 7 and 15). To correlate elements, a join
20
i3QL operatorsOperator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERENOT EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1UNION SELECT ... FROM t2
Union all SELECT ... FROM t1UNION ALL SELECT ... FROM t2
Intersection SELECT ... FROM t1INTERSECT SELECT ... FROM t2
Table 1: i3QL operators.
The basis of the change propagation is the Observer inter-face depicted in Figure 3. The interface declares three atomicchange events for addition, removal and update.
1 trait Observer[V] {2 def added(v: V)3 def removed(v: V)4 def updated(oldV: V, newV: V)5 }
Figure 3: The observer interface for change propagation.
Every i3QL operator implements the Observer trait and acorresponding Observable trait. The implementation of theObserver defines incremental semantics of an operator. Anoperator registers itself as an observer of its predecessors inthe graph and receives atomic change events through any ofthe three methods of Observer.
Selection Operator To illustrate the implementation of op-erators, we show the implementation of the selection oper-ator in Figure 4. The selection operator observes an under-lying relation containing elements of type V and filters theelements using a predicate function. Each operator has a type
for resulting elements, defined by the type argument passedto Observable. In case of selections,V is the input and outputtype of the operator. The change propagation of selection iscomparatively simple: Selection only propagates a changeto its observers if the involved value satisfies the predicate.Finally, selection registers itself as an observer of the under-lying relation.
1 class Selection[V](rel: Relation[V], predicate: V => Boolean)2 extends Observer[V] with Observable[V] {3 def added(v: V) { if(predicate(v)) notify added(v) }4 def removed(v: V) { if(predicate(v)) notify removed(v) }5 def updated(oldV: V, newV: V) {6 if(predicate(oldV)) {7 if(predicate(newV)) notify updated(oldV, newV)8 else notify removed(oldV) }9 else if(predicate(newV)) notify added(newV) }
10 rel addObserver this11 }
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator thatcorrelates elements of a left-hand and right-hand side rela-tion. The results are pairs of elements from the two rela-tions. Instead of a binary predicate expressing the join con-dition, our join operator uses two functions that map ele-ments to key values; the operator joins two elements if theyare mapped to the same key. For example, in the join exam-ple shown in Sec. 2, we compared the properties firstName
and lastName of elements from two relations. i3QL automat-ically translates this condition into two key functions of theform (s:Student) => (s.firstName, s.lastName), where the key isa tuple of all values that have to be equal.
1 class Join[V1, V2, Key](2 left: Relation[V1], right: Relation[V2],3 leftKeyFun: V1 => Key, rightKeyFun: V2 => Key)4 extends Observable[(V1,V2)] {5 val leftIndex : Map[Key, List[V1]] = ...6 val rightIndex : Map[Key, List[V2]] = ...7 object LeftObserver extends Observer[V1] {8 def added(v: V1) {9 val key = leftKeyFun(v)
10 rightIndex.get(key) match {11 case Some(list) =>12 list.foreach( u => notify added ((u,v)) )13 case => // no changes propagated14 }}...}15 object RightObserver extends Observer[V2] { ... }16 }
Figure 5: Excerpt of the incremental join operator.
Figure 5 shows an excerpt of the implementation of the in-cremental join operator. The join—and any binary operatorin general—observes the input relations via two differentlytyped observers (line 7 and 15). To correlate elements, a join
Operator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERENOT EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1UNION SELECT ... FROM t2
Union all SELECT ... FROM t1UNION ALL SELECT ... FROM t2
Intersection SELECT ... FROM t1INTERSECT SELECT ... FROM t2
Table 1: i3QL operators.
The basis of the change propagation is the Observer inter-face depicted in Figure 3. The interface declares three atomicchange events for addition, removal and update.
1 trait Observer[V] {2 def added(v: V)3 def removed(v: V)4 def updated(oldV: V, newV: V)5 }
Figure 3: The observer interface for change propagation.
Every i3QL operator implements the Observer trait and acorresponding Observable trait. The implementation of theObserver defines incremental semantics of an operator. Anoperator registers itself as an observer of its predecessors inthe graph and receives atomic change events through any ofthe three methods of Observer.
Selection Operator To illustrate the implementation of op-erators, we show the implementation of the selection oper-ator in Figure 4. The selection operator observes an under-lying relation containing elements of type V and filters theelements using a predicate function. Each operator has a type
for resulting elements, defined by the type argument passedto Observable. In case of selections,V is the input and outputtype of the operator. The change propagation of selection iscomparatively simple: Selection only propagates a changeto its observers if the involved value satisfies the predicate.Finally, selection registers itself as an observer of the under-lying relation.
1 class Selection[V](rel: Relation[V], predicate: V => Boolean)2 extends Observer[V] with Observable[V] {3 def added(v: V) { if(predicate(v)) notify added(v) }4 def removed(v: V) { if(predicate(v)) notify removed(v) }5 def updated(oldV: V, newV: V) {6 if(predicate(oldV)) {7 if(predicate(newV)) notify updated(oldV, newV)8 else notify removed(oldV) }9 else if(predicate(newV)) notify added(newV) }
10 rel addObserver this11 }
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator thatcorrelates elements of a left-hand and right-hand side rela-tion. The results are pairs of elements from the two rela-tions. Instead of a binary predicate expressing the join con-dition, our join operator uses two functions that map ele-ments to key values; the operator joins two elements if theyare mapped to the same key. For example, in the join exam-ple shown in Sec. 2, we compared the properties firstName
and lastName of elements from two relations. i3QL automat-ically translates this condition into two key functions of theform (s:Student) => (s.firstName, s.lastName), where the key isa tuple of all values that have to be equal.
1 class Join[V1, V2, Key](2 left: Relation[V1], right: Relation[V2],3 leftKeyFun: V1 => Key, rightKeyFun: V2 => Key)4 extends Observable[(V1,V2)] {5 val leftIndex : Map[Key, List[V1]] = ...6 val rightIndex : Map[Key, List[V2]] = ...7 object LeftObserver extends Observer[V1] {8 def added(v: V1) {9 val key = leftKeyFun(v)
10 rightIndex.get(key) match {11 case Some(list) =>12 list.foreach( u => notify added ((u,v)) )13 case => // no changes propagated14 }}...}15 object RightObserver extends Observer[V2] { ... }16 }
Figure 5: Excerpt of the incremental join operator.
Figure 5 shows an excerpt of the implementation of the in-cremental join operator. The join—and any binary operatorin general—observes the input relations via two differentlytyped observers (line 7 and 15). To correlate elements, a join
Support recursion WITH RECURSIVE (init UNION ALL query)
20
i3QL operatorsOperator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERENOT EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1UNION SELECT ... FROM t2
Union all SELECT ... FROM t1UNION ALL SELECT ... FROM t2
Intersection SELECT ... FROM t1INTERSECT SELECT ... FROM t2
Table 1: i3QL operators.
The basis of the change propagation is the Observer inter-face depicted in Figure 3. The interface declares three atomicchange events for addition, removal and update.
1 trait Observer[V] {2 def added(v: V)3 def removed(v: V)4 def updated(oldV: V, newV: V)5 }
Figure 3: The observer interface for change propagation.
Every i3QL operator implements the Observer trait and acorresponding Observable trait. The implementation of theObserver defines incremental semantics of an operator. Anoperator registers itself as an observer of its predecessors inthe graph and receives atomic change events through any ofthe three methods of Observer.
Selection Operator To illustrate the implementation of op-erators, we show the implementation of the selection oper-ator in Figure 4. The selection operator observes an under-lying relation containing elements of type V and filters theelements using a predicate function. Each operator has a type
for resulting elements, defined by the type argument passedto Observable. In case of selections,V is the input and outputtype of the operator. The change propagation of selection iscomparatively simple: Selection only propagates a changeto its observers if the involved value satisfies the predicate.Finally, selection registers itself as an observer of the under-lying relation.
1 class Selection[V](rel: Relation[V], predicate: V => Boolean)2 extends Observer[V] with Observable[V] {3 def added(v: V) { if(predicate(v)) notify added(v) }4 def removed(v: V) { if(predicate(v)) notify removed(v) }5 def updated(oldV: V, newV: V) {6 if(predicate(oldV)) {7 if(predicate(newV)) notify updated(oldV, newV)8 else notify removed(oldV) }9 else if(predicate(newV)) notify added(newV) }
10 rel addObserver this11 }
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator thatcorrelates elements of a left-hand and right-hand side rela-tion. The results are pairs of elements from the two rela-tions. Instead of a binary predicate expressing the join con-dition, our join operator uses two functions that map ele-ments to key values; the operator joins two elements if theyare mapped to the same key. For example, in the join exam-ple shown in Sec. 2, we compared the properties firstName
and lastName of elements from two relations. i3QL automat-ically translates this condition into two key functions of theform (s:Student) => (s.firstName, s.lastName), where the key isa tuple of all values that have to be equal.
1 class Join[V1, V2, Key](2 left: Relation[V1], right: Relation[V2],3 leftKeyFun: V1 => Key, rightKeyFun: V2 => Key)4 extends Observable[(V1,V2)] {5 val leftIndex : Map[Key, List[V1]] = ...6 val rightIndex : Map[Key, List[V2]] = ...7 object LeftObserver extends Observer[V1] {8 def added(v: V1) {9 val key = leftKeyFun(v)
10 rightIndex.get(key) match {11 case Some(list) =>12 list.foreach( u => notify added ((u,v)) )13 case => // no changes propagated14 }}...}15 object RightObserver extends Observer[V2] { ... }16 }
Figure 5: Excerpt of the incremental join operator.
Figure 5 shows an excerpt of the implementation of the in-cremental join operator. The join—and any binary operatorin general—observes the input relations via two differentlytyped observers (line 7 and 15). To correlate elements, a join
Operator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERENOT EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1UNION SELECT ... FROM t2
Union all SELECT ... FROM t1UNION ALL SELECT ... FROM t2
Intersection SELECT ... FROM t1INTERSECT SELECT ... FROM t2
Table 1: i3QL operators.
The basis of the change propagation is the Observer inter-face depicted in Figure 3. The interface declares three atomicchange events for addition, removal and update.
1 trait Observer[V] {2 def added(v: V)3 def removed(v: V)4 def updated(oldV: V, newV: V)5 }
Figure 3: The observer interface for change propagation.
Every i3QL operator implements the Observer trait and acorresponding Observable trait. The implementation of theObserver defines incremental semantics of an operator. Anoperator registers itself as an observer of its predecessors inthe graph and receives atomic change events through any ofthe three methods of Observer.
Selection Operator To illustrate the implementation of op-erators, we show the implementation of the selection oper-ator in Figure 4. The selection operator observes an under-lying relation containing elements of type V and filters theelements using a predicate function. Each operator has a type
for resulting elements, defined by the type argument passedto Observable. In case of selections,V is the input and outputtype of the operator. The change propagation of selection iscomparatively simple: Selection only propagates a changeto its observers if the involved value satisfies the predicate.Finally, selection registers itself as an observer of the under-lying relation.
1 class Selection[V](rel: Relation[V], predicate: V => Boolean)2 extends Observer[V] with Observable[V] {3 def added(v: V) { if(predicate(v)) notify added(v) }4 def removed(v: V) { if(predicate(v)) notify removed(v) }5 def updated(oldV: V, newV: V) {6 if(predicate(oldV)) {7 if(predicate(newV)) notify updated(oldV, newV)8 else notify removed(oldV) }9 else if(predicate(newV)) notify added(newV) }
10 rel addObserver this11 }
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator thatcorrelates elements of a left-hand and right-hand side rela-tion. The results are pairs of elements from the two rela-tions. Instead of a binary predicate expressing the join con-dition, our join operator uses two functions that map ele-ments to key values; the operator joins two elements if theyare mapped to the same key. For example, in the join exam-ple shown in Sec. 2, we compared the properties firstName
and lastName of elements from two relations. i3QL automat-ically translates this condition into two key functions of theform (s:Student) => (s.firstName, s.lastName), where the key isa tuple of all values that have to be equal.
1 class Join[V1, V2, Key](2 left: Relation[V1], right: Relation[V2],3 leftKeyFun: V1 => Key, rightKeyFun: V2 => Key)4 extends Observable[(V1,V2)] {5 val leftIndex : Map[Key, List[V1]] = ...6 val rightIndex : Map[Key, List[V2]] = ...7 object LeftObserver extends Observer[V1] {8 def added(v: V1) {9 val key = leftKeyFun(v)
10 rightIndex.get(key) match {11 case Some(list) =>12 list.foreach( u => notify added ((u,v)) )13 case => // no changes propagated14 }}...}15 object RightObserver extends Observer[V2] { ... }16 }
Figure 5: Excerpt of the incremental join operator.
Figure 5 shows an excerpt of the implementation of the in-cremental join operator. The join—and any binary operatorin general—observes the input relations via two differentlytyped observers (line 7 and 15). To correlate elements, a join
Support recursion WITH RECURSIVE (init UNION ALL query)Extensible ! Add new operators! Add new optimizations
20
i3QL operatorsOperator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERENOT EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1UNION SELECT ... FROM t2
Union all SELECT ... FROM t1UNION ALL SELECT ... FROM t2
Intersection SELECT ... FROM t1INTERSECT SELECT ... FROM t2
Table 1: i3QL operators.
The basis of the change propagation is the Observer inter-face depicted in Figure 3. The interface declares three atomicchange events for addition, removal and update.
1 trait Observer[V] {2 def added(v: V)3 def removed(v: V)4 def updated(oldV: V, newV: V)5 }
Figure 3: The observer interface for change propagation.
Every i3QL operator implements the Observer trait and acorresponding Observable trait. The implementation of theObserver defines incremental semantics of an operator. Anoperator registers itself as an observer of its predecessors inthe graph and receives atomic change events through any ofthe three methods of Observer.
Selection Operator To illustrate the implementation of op-erators, we show the implementation of the selection oper-ator in Figure 4. The selection operator observes an under-lying relation containing elements of type V and filters theelements using a predicate function. Each operator has a type
for resulting elements, defined by the type argument passedto Observable. In case of selections,V is the input and outputtype of the operator. The change propagation of selection iscomparatively simple: Selection only propagates a changeto its observers if the involved value satisfies the predicate.Finally, selection registers itself as an observer of the under-lying relation.
1 class Selection[V](rel: Relation[V], predicate: V => Boolean)2 extends Observer[V] with Observable[V] {3 def added(v: V) { if(predicate(v)) notify added(v) }4 def removed(v: V) { if(predicate(v)) notify removed(v) }5 def updated(oldV: V, newV: V) {6 if(predicate(oldV)) {7 if(predicate(newV)) notify updated(oldV, newV)8 else notify removed(oldV) }9 else if(predicate(newV)) notify added(newV) }
10 rel addObserver this11 }
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator thatcorrelates elements of a left-hand and right-hand side rela-tion. The results are pairs of elements from the two rela-tions. Instead of a binary predicate expressing the join con-dition, our join operator uses two functions that map ele-ments to key values; the operator joins two elements if theyare mapped to the same key. For example, in the join exam-ple shown in Sec. 2, we compared the properties firstName
and lastName of elements from two relations. i3QL automat-ically translates this condition into two key functions of theform (s:Student) => (s.firstName, s.lastName), where the key isa tuple of all values that have to be equal.
1 class Join[V1, V2, Key](2 left: Relation[V1], right: Relation[V2],3 leftKeyFun: V1 => Key, rightKeyFun: V2 => Key)4 extends Observable[(V1,V2)] {5 val leftIndex : Map[Key, List[V1]] = ...6 val rightIndex : Map[Key, List[V2]] = ...7 object LeftObserver extends Observer[V1] {8 def added(v: V1) {9 val key = leftKeyFun(v)
10 rightIndex.get(key) match {11 case Some(list) =>12 list.foreach( u => notify added ((u,v)) )13 case => // no changes propagated14 }}...}15 object RightObserver extends Observer[V2] { ... }16 }
Figure 5: Excerpt of the incremental join operator.
Figure 5 shows an excerpt of the implementation of the in-cremental join operator. The join—and any binary operatorin general—observes the input relations via two differentlytyped observers (line 7 and 15). To correlate elements, a join
Operator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERENOT EXISTS(SELECT ... FROM t2WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1UNION SELECT ... FROM t2
Union all SELECT ... FROM t1UNION ALL SELECT ... FROM t2
Intersection SELECT ... FROM t1INTERSECT SELECT ... FROM t2
Table 1: i3QL operators.
The basis of the change propagation is the Observer inter-face depicted in Figure 3. The interface declares three atomicchange events for addition, removal and update.
1 trait Observer[V] {2 def added(v: V)3 def removed(v: V)4 def updated(oldV: V, newV: V)5 }
Figure 3: The observer interface for change propagation.
Every i3QL operator implements the Observer trait and acorresponding Observable trait. The implementation of theObserver defines incremental semantics of an operator. Anoperator registers itself as an observer of its predecessors inthe graph and receives atomic change events through any ofthe three methods of Observer.
Selection Operator To illustrate the implementation of op-erators, we show the implementation of the selection oper-ator in Figure 4. The selection operator observes an under-lying relation containing elements of type V and filters theelements using a predicate function. Each operator has a type
for resulting elements, defined by the type argument passedto Observable. In case of selections,V is the input and outputtype of the operator. The change propagation of selection iscomparatively simple: Selection only propagates a changeto its observers if the involved value satisfies the predicate.Finally, selection registers itself as an observer of the under-lying relation.
1 class Selection[V](rel: Relation[V], predicate: V => Boolean)2 extends Observer[V] with Observable[V] {3 def added(v: V) { if(predicate(v)) notify added(v) }4 def removed(v: V) { if(predicate(v)) notify removed(v) }5 def updated(oldV: V, newV: V) {6 if(predicate(oldV)) {7 if(predicate(newV)) notify updated(oldV, newV)8 else notify removed(oldV) }9 else if(predicate(newV)) notify added(newV) }
10 rel addObserver this11 }
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator thatcorrelates elements of a left-hand and right-hand side rela-tion. The results are pairs of elements from the two rela-tions. Instead of a binary predicate expressing the join con-dition, our join operator uses two functions that map ele-ments to key values; the operator joins two elements if theyare mapped to the same key. For example, in the join exam-ple shown in Sec. 2, we compared the properties firstName
and lastName of elements from two relations. i3QL automat-ically translates this condition into two key functions of theform (s:Student) => (s.firstName, s.lastName), where the key isa tuple of all values that have to be equal.
1 class Join[V1, V2, Key](2 left: Relation[V1], right: Relation[V2],3 leftKeyFun: V1 => Key, rightKeyFun: V2 => Key)4 extends Observable[(V1,V2)] {5 val leftIndex : Map[Key, List[V1]] = ...6 val rightIndex : Map[Key, List[V2]] = ...7 object LeftObserver extends Observer[V1] {8 def added(v: V1) {9 val key = leftKeyFun(v)
10 rightIndex.get(key) match {11 case Some(list) =>12 list.foreach( u => notify added ((u,v)) )13 case => // no changes propagated14 }}...}15 object RightObserver extends Observer[V2] { ... }16 }
Figure 5: Excerpt of the incremental join operator.
Figure 5 shows an excerpt of the implementation of the in-cremental join operator. The join—and any binary operatorin general—observes the input relations via two differentlytyped observers (line 7 and 15). To correlate elements, a join
Support recursion WITH RECURSIVE (init UNION ALL query)Extensible ! Add new operators! Add new optimizations
Use regular Scala code for nonincremental subcomputations
21
Performance evaluation
22
Implemented 15 FindBugsJVM bytecode analyses in i3QL
UpdateChange
22
Implemented 15 FindBugsJVM bytecode analyses in i3QL
Test data: Vespucci revision history! 381 class files initially! 242 changes (add, remove, update)! 1 change affects 1 to 312 class filesUpdateChange
22
Implemented 15 FindBugsJVM bytecode analyses in i3QL
Test data: Vespucci revision history! 381 class files initially! 242 changes (add, remove, update)! 1 change affects 1 to 312 class files
For each analysis! Analyze initial 381 class files! Replay changes in order
UpdateChange
23
0 50 100 150 200 250 300
050
100
150
200
250
Change size (in number of affected classes)
Tim
e (m
s)
(a) Update time per change.
0 50 100 150 200 250 300
−100
−50
050
100
Change size (in number of affected classes)
Mem
ory
chan
ge (K
iB)
(b) Variation in memory consumption per change.
Figure 13: Mean time and memory consumption per changefor all 15 Findbugs analyses implemented with i3QL.
ory consumption averaged over all 15 analyses (the standarddeviation was low). The benchmarks have been performed 5times for each analysis with an additional 4 warmup itera-tions. The end result for each analysis is the average of those5 measurements.
Figure 13a shows the time it takes to update the analysisresult for each change, that is all classes that are changed si-multaneously. The x-axis shows the size of a change and they-axis shows the time that is needed to update the analysisresult. We ordered data points according to the change size.The plot shows that the time it takes to update the overallanalysis result is roughly linear in the change size; it doesnot seem to depend on the history of changes already pro-cessed or on the size of the initial revision.
A comparison with the non-incremental analyses im-plemented in FindBugs unsurprisingly shows an order-of-magnitude improvement in run-time performance. Runningan analysis in FindBugs for the initial revision takes 534 mson average, whereas i3QL only takes 129 ms on average.Since the FindBugs implementation will take roughly the
same time for each revision, checking all revisions amountsto about 130 seconds, whereas i3QL only requires little morethan 2 seconds. This means that on our test data i3QL pro-vides a speedup of 65x compared to the non-incrementalFindBugs analyses.
Figure 13b shows the variation in memory for eachchange. Like in the graphic above, the x-axis displays thesize of one change. The y-axis shows how much more (orless) memory is needed after the change has been processed.We run the garbage collector after the replay and propaga-tion of each change to get accurate measurements. The plotshows that the memory impact of a change is moderate (atmost 120 KiB for a change affecting more than 250 classes),which means that auxiliary data stored to enable incremen-talization does not have a significant effect. For comparison,the class files of the initial revision account for 1.39 MiB.
Effect of Optimizations. To measure the effect of opti-mizations in our implementation, we run the analyses in twoconfigurations of i3QL: with and without the optimizationspresented in Section 4. The results are shown in Table 2. Therows of the table are indexed by the analyses (A through O).The execution times of the analyses with i3QL optimizationsturned off respectively on are shown in the first respectivelythe second column of the table. The last column shows thespeedup achieved by turning optimization on.
By comparing the results, we observe that in 6 analy-ses, the optimizations improved performance significantly(> 20%), most prominently for analyses A, C, D, and E.In the other analyses, the effect of optimization is less sig-nificant. In case of analysis B, the optimizations even hada small negative effect. The reason for such a difference istwofold.
First, analyses A, C, D, E, F, G, K, N contain queries overthe byte-code instruction of the codebase—a much largertable than the table of method declarations, for example.Thus, the optimizations have a greater effect. Second, A,C and D use equi-joins, which further amplifies this effect.Equi-joins are only created with optimizations; without op-timizations those relations are simply cross products and se-lections. Equi-joins are very benefiting for large relations(e.g. instructions), since the join operator hashes its values—opposed to cross product and selection, which have to iter-ate over all elements in order to find equal elements. Thedifference between analysis A and C, D is that analysis Ajoins two large instruction relations while C and D join aninstruction relation with other relations that are not as large.
Summary. Our performance evaluation shows that i3QLqueries run an order of magnitude faster than their non-incremental counterparts, without requiring much memory.The run time for a change is linear in the change size andindependent of size of the initial revision of the input. Theevaluation clearly shows that optimizations bring significantperformance improvements; especially, they ensure that theanalyses execution time scales well for big sets of input data.
Update time per change
Mean update time for all 15 analyses
23
0 50 100 150 200 250 300
050
100
150
200
250
Change size (in number of affected classes)
Tim
e (m
s)
(a) Update time per change.
0 50 100 150 200 250 300
−100
−50
050
100
Change size (in number of affected classes)
Mem
ory
chan
ge (K
iB)
(b) Variation in memory consumption per change.
Figure 13: Mean time and memory consumption per changefor all 15 Findbugs analyses implemented with i3QL.
ory consumption averaged over all 15 analyses (the standarddeviation was low). The benchmarks have been performed 5times for each analysis with an additional 4 warmup itera-tions. The end result for each analysis is the average of those5 measurements.
Figure 13a shows the time it takes to update the analysisresult for each change, that is all classes that are changed si-multaneously. The x-axis shows the size of a change and they-axis shows the time that is needed to update the analysisresult. We ordered data points according to the change size.The plot shows that the time it takes to update the overallanalysis result is roughly linear in the change size; it doesnot seem to depend on the history of changes already pro-cessed or on the size of the initial revision.
A comparison with the non-incremental analyses im-plemented in FindBugs unsurprisingly shows an order-of-magnitude improvement in run-time performance. Runningan analysis in FindBugs for the initial revision takes 534 mson average, whereas i3QL only takes 129 ms on average.Since the FindBugs implementation will take roughly the
same time for each revision, checking all revisions amountsto about 130 seconds, whereas i3QL only requires little morethan 2 seconds. This means that on our test data i3QL pro-vides a speedup of 65x compared to the non-incrementalFindBugs analyses.
Figure 13b shows the variation in memory for eachchange. Like in the graphic above, the x-axis displays thesize of one change. The y-axis shows how much more (orless) memory is needed after the change has been processed.We run the garbage collector after the replay and propaga-tion of each change to get accurate measurements. The plotshows that the memory impact of a change is moderate (atmost 120 KiB for a change affecting more than 250 classes),which means that auxiliary data stored to enable incremen-talization does not have a significant effect. For comparison,the class files of the initial revision account for 1.39 MiB.
Effect of Optimizations. To measure the effect of opti-mizations in our implementation, we run the analyses in twoconfigurations of i3QL: with and without the optimizationspresented in Section 4. The results are shown in Table 2. Therows of the table are indexed by the analyses (A through O).The execution times of the analyses with i3QL optimizationsturned off respectively on are shown in the first respectivelythe second column of the table. The last column shows thespeedup achieved by turning optimization on.
By comparing the results, we observe that in 6 analy-ses, the optimizations improved performance significantly(> 20%), most prominently for analyses A, C, D, and E.In the other analyses, the effect of optimization is less sig-nificant. In case of analysis B, the optimizations even hada small negative effect. The reason for such a difference istwofold.
First, analyses A, C, D, E, F, G, K, N contain queries overthe byte-code instruction of the codebase—a much largertable than the table of method declarations, for example.Thus, the optimizations have a greater effect. Second, A,C and D use equi-joins, which further amplifies this effect.Equi-joins are only created with optimizations; without op-timizations those relations are simply cross products and se-lections. Equi-joins are very benefiting for large relations(e.g. instructions), since the join operator hashes its values—opposed to cross product and selection, which have to iter-ate over all elements in order to find equal elements. Thedifference between analysis A and C, D is that analysis Ajoins two large instruction relations while C and D join aninstruction relation with other relations that are not as large.
Summary. Our performance evaluation shows that i3QLqueries run an order of magnitude faster than their non-incremental counterparts, without requiring much memory.The run time for a change is linear in the change size andindependent of size of the initial revision of the input. Theevaluation clearly shows that optimizations bring significantperformance improvements; especially, they ensure that theanalyses execution time scales well for big sets of input data.
Update time per change
Update time ~ linear in change size!! ! Small change => small effect! ! Larger change => larger effect
Mean update time for all 15 analyses
23
0 50 100 150 200 250 300
050
100
150
200
250
Change size (in number of affected classes)
Tim
e (m
s)
(a) Update time per change.
0 50 100 150 200 250 300
−100
−50
050
100
Change size (in number of affected classes)
Mem
ory
chan
ge (K
iB)
(b) Variation in memory consumption per change.
Figure 13: Mean time and memory consumption per changefor all 15 Findbugs analyses implemented with i3QL.
ory consumption averaged over all 15 analyses (the standarddeviation was low). The benchmarks have been performed 5times for each analysis with an additional 4 warmup itera-tions. The end result for each analysis is the average of those5 measurements.
Figure 13a shows the time it takes to update the analysisresult for each change, that is all classes that are changed si-multaneously. The x-axis shows the size of a change and they-axis shows the time that is needed to update the analysisresult. We ordered data points according to the change size.The plot shows that the time it takes to update the overallanalysis result is roughly linear in the change size; it doesnot seem to depend on the history of changes already pro-cessed or on the size of the initial revision.
A comparison with the non-incremental analyses im-plemented in FindBugs unsurprisingly shows an order-of-magnitude improvement in run-time performance. Runningan analysis in FindBugs for the initial revision takes 534 mson average, whereas i3QL only takes 129 ms on average.Since the FindBugs implementation will take roughly the
same time for each revision, checking all revisions amountsto about 130 seconds, whereas i3QL only requires little morethan 2 seconds. This means that on our test data i3QL pro-vides a speedup of 65x compared to the non-incrementalFindBugs analyses.
Figure 13b shows the variation in memory for eachchange. Like in the graphic above, the x-axis displays thesize of one change. The y-axis shows how much more (orless) memory is needed after the change has been processed.We run the garbage collector after the replay and propaga-tion of each change to get accurate measurements. The plotshows that the memory impact of a change is moderate (atmost 120 KiB for a change affecting more than 250 classes),which means that auxiliary data stored to enable incremen-talization does not have a significant effect. For comparison,the class files of the initial revision account for 1.39 MiB.
Effect of Optimizations. To measure the effect of opti-mizations in our implementation, we run the analyses in twoconfigurations of i3QL: with and without the optimizationspresented in Section 4. The results are shown in Table 2. Therows of the table are indexed by the analyses (A through O).The execution times of the analyses with i3QL optimizationsturned off respectively on are shown in the first respectivelythe second column of the table. The last column shows thespeedup achieved by turning optimization on.
By comparing the results, we observe that in 6 analy-ses, the optimizations improved performance significantly(> 20%), most prominently for analyses A, C, D, and E.In the other analyses, the effect of optimization is less sig-nificant. In case of analysis B, the optimizations even hada small negative effect. The reason for such a difference istwofold.
First, analyses A, C, D, E, F, G, K, N contain queries overthe byte-code instruction of the codebase—a much largertable than the table of method declarations, for example.Thus, the optimizations have a greater effect. Second, A,C and D use equi-joins, which further amplifies this effect.Equi-joins are only created with optimizations; without op-timizations those relations are simply cross products and se-lections. Equi-joins are very benefiting for large relations(e.g. instructions), since the join operator hashes its values—opposed to cross product and selection, which have to iter-ate over all elements in order to find equal elements. Thedifference between analysis A and C, D is that analysis Ajoins two large instruction relations while C and D join aninstruction relation with other relations that are not as large.
Summary. Our performance evaluation shows that i3QLqueries run an order of magnitude faster than their non-incremental counterparts, without requiring much memory.The run time for a change is linear in the change size andindependent of size of the initial revision of the input. Theevaluation clearly shows that optimizations bring significantperformance improvements; especially, they ensure that theanalyses execution time scales well for big sets of input data.
Update time per change
Update time ~ linear in change size!! ! Small change => small effect! ! Larger change => larger effect
Mean update time for all 15 analyses
Comparison incremental vs nonincrementalFindBugs
i3QL sequentiali3QL incremental
1 10 100time (seconds), log-scale
130s31s
2s
24
Effect of optimizationstim
e (s
econ
ds),
log-
scal
e
1
10
100
1000
A B C D E F G H I J K L M N O
i3QL without optimizations i3QL with optimizations
24
Effect of optimizationstim
e (s
econ
ds),
log-
scal
e
1
10
100
1000
A B C D E F G H I J K L M N O
i3QL without optimizations i3QL with optimizations
Speedup up to 575xSlowdown of 4% in one case
24
Effect of optimizationstim
e (s
econ
ds),
log-
scal
e
1
10
100
1000
A B C D E F G H I J K L M N O
i3QL without optimizations i3QL with optimizations
Speedup up to 575xSlowdown of 4% in one case
When optimizations apply, huge effect
25
In the paper
• Detailed description of incremental operators
25
In the paper
• Detailed description of incremental operators
• Case study:Complete definition of incremental parser
25
In the paper
• Detailed description of incremental operators
• Case study:Complete definition of incremental parser
• Evaluation of memory consumption
25
In the paper
• Detailed description of incremental operators
• Case study:Complete definition of incremental parser
• Evaluation of memory consumption
• Detailed discussion of related work!- embedded query languages!- in-memory collections!- constraints and functional dependencies!- view maintenance in DB systems!- optimization techniques
i3QL: Language-Integrated Live Data Views
�������
��� ������� �
����
��������������
��
����� �
����������
�������
�������� �
���
integrated!! ! ! !in-memory! ! ! ! !incremental!! ! high-performance applicable
Scala-embedded DSL!collections API!based on relational algebrausing algebraic laws and partial evaluation used for FindBugs analyses on JVM byte code
https://github.com/seba--/i3QL/