LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development...
-
Upload
gwen-hopkins -
Category
Documents
-
view
217 -
download
0
Transcript of LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development...
![Page 1: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/1.jpg)
LINQ, Take TwoRealizing the LINQ to Everything Dream
Bart J.F. De SmetSenior Software Development EngineerMicrosoft Corporation
![Page 2: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/2.jpg)
A Historical Perspective
Censored
7 years ago
Little recent innovation
Where’s the cloud?
![Page 3: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/3.jpg)
Why?
What?How?
Language Integrated QueryMonads
![Page 4: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/4.jpg)
IEnumerable<T>
IQueryable<T>
new[]{ 42 }
SelectMany
IEnumerable<R> SelectMany<T, R>( this IEnumerable<T> source, Func<T, IEnumerable<R>> selector)
Could there be more?
Also see www.codeplex.com/LINQSQO for “Project MinLINQ”
The Monadic Bind Operator Revealed
![Page 5: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/5.jpg)
from _ in name from s in _.ToUpper() select s
name.SelectMany( _ => _.ToUpper(), s => s)Compiler
Null-propagating dot
string s = name?.ToUpper();
Syntactic sugar
One single library function suffices
Building the Maybe MonadFor Fun and No (?) Profit
![Page 6: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/6.jpg)
demo
Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team
Essential LINQ
![Page 7: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/7.jpg)
var src = new Source<Product>();
var res = from p in src where p.Price > 100m group p by p.Category;
foreach (var p in res) Console.WriteLine(p);
ImplementsIQueryable<T
>
Compiles fine
Does it really have to be a runtime check?
Query Providers RevisitedDo We Need IQueryable<T>?
![Page 8: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/8.jpg)
var res = from p in src
where p.Price > 100m
group p by p.Category;
var res = src
.Where(p => p.Price > 100m)
.GroupBy(p => p.Category);
Syntactic sugar
Can be instance methods
Source<T> Filtered<T> Projected<T>Where Select
No GroupBy “edge”
Leveraging The Query Pattern
![Page 9: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/9.jpg)
var res = from tweet in twitter where tweet.
AboutFromLocation
== “Bart”From
From
AboutLocation
== “LINQ”About
where tweet. About
Query “learns”
class Twitter{ public TwitterByFrom Where(Func<TweetAboutFromLoc, FilterFrom> filter);
// Other filter methods}
Recipe
Method overloadsType state machineOperator overloads
“Has a” type
From operator overloadingclass TwitterByFrom
{ public TwitterByAboutFrom Where(Func<TweetAboutLoc, FilterAbout> filter);
// Other filter methods // Fields with current filters}
select tweet;
Custom syntax trees
Taking It One Step Further
class TweetAboutFromLoc{ public FromString From; public AboutString About; public LocString Location;}
class FromString{ FilterFrom operator ==( FromString f, string s)}
class TweetAboutLoc{ public AboutString About; public LocString Location;}
class AboutString{ FilterAbout operator ==( AboutString f, string s)}
![Page 10: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/10.jpg)
demo
Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team
Query Providers Revisited
![Page 11: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/11.jpg)
Bottom-Up Tree RewritingA rule drive query provider
+
1*
2 3
Source domain
Rule Set
Patterns with rewrite actions
Target domain
BURS Tree Rewriting Engine
E.g. LINQ Expression Trees E.g. T-SQL Expression Trees
Const(2) Const(3)
Mul
Inc
Currently built at runtime, e.g. once per LINQ provider
that gets loaded
![Page 12: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/12.jpg)
x1 $
Const(x)
State TablesLeaf node
? : int 1
Final states
1 1 $
Any source leaf node with an int constant value can be
reduced using rule 1.
Final states have a cost (incremental, see further).
![Page 13: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/13.jpg)
x
+
a b
1 $Const(x)
2 $Add(a,b)
State TablesLeaf node
? : int 1
Wildcard
a : int 2 *
+ 2 *
2 * 3
Final states
1 1 $
3 2 $
Wildcard b doesn’t get its own entry, since it has the same type as wildcard a.
This reduces space in the BURS tables.
2D table for state transition of the +
operator.
Wildcard matches any subtree that’s assignable to int.
![Page 14: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/14.jpg)
x
+
a b
*
a b
1 $Const(x)
2 $Add(a,b)
3 $Mul(a,b)
State TablesLeaf node
? : int 1
Wildcard
a : int 2 *
+ 2 *
2 * 3
* 2 *
2 * 4
Final states
1 1 $
3 2 $
4 3 $
What’s the meaning of costs?
The cost of 3$ is incremental and has to be added to the cost of all the wildcard matches.
In this case, a match of the Mul-rule (4 ) costs 3$ + costa + costb.
![Page 15: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/15.jpg)
x
+
a b
*
a b
+
c*
a b
1 $Const(x)
2 $Add(a,b)
3 $Mul(a,b)
4 $AddMul(a,b,c)
State TablesLeaf node
? : int 1
Wildcard
a : int 2 *
+ 2 *
2 * 3
4 5
* 2 *
2 * 4
Final states
1 1 $
3 2 $
4 3 $
5 4 $
A pattern forest is kept to map
subtrees on already existent states. Here (a * b) matches state 4.
![Page 16: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/16.jpg)
x
+
a b
*
a b
+
c*
a b
+
a 1
1 $Const(x)
2 $Add(a,b)
3 $Mul(a,b)
4 $AddMul(a,b,c)
1 $Inc(a)
State TablesLeaf node
? : int 1
1 6
Wildcard
a : int 2 *
+ 2 * 6
2 * 3 7
4 5
* 2 *
2 * 4
Final states
1 1 $
3 2 $
4 3 $
5 4 $
7 1 $
![Page 17: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/17.jpg)
Subject Tree Coverings
2 3
* 1
+
Leaf node
? : int 1
1 6
Wildcard
a : int 2 *
+ 2 * 6
2 * 3 7
4 5
* 2 *
2 * 4
Final states
1 1 $
3 2 $
4 3 $
5 4 $
7 1 $
1 $ 1 $
5 $
6 $Constant leaf node 1
doesn’t contribute cost (part of pattern).
![Page 18: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/18.jpg)
demo
Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team
Bottom Up Rewriting
![Page 19: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/19.jpg)
Environment
MoveN
ext
Got next?
Application
On
Next
Have next!
IEnumerable<T>IEnumerator<T>
IObservable<T>IObserver<T>
Inte
racti
ve R
eactiv
eData Retrieval – Push or Pull?
![Page 20: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/20.jpg)
Event Streams
Towards a unified programming modelProducers are observable sequences
.NET events, WinRT events, sensor APIs, APM methods, tasks, etc.
Consumers are observersHooking up “continuations” or handlers
Observable
Subscribe
Observer
![Page 21: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/21.jpg)
Event Stream Processing Interfacesnamespace System{ public interface IObservable<out T> { IDisposable Subscribe(IObserver<T> observer); }
public interface IObserver<in T> { void OnNext(T value); void OnError(Exception error); void OnCompleted(); }}
![Page 22: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/22.jpg)
Fix
ed
(MS
IL)
Tran
sla
tab
le
(Exp
ressio
n
trees)
ToQueryable
ToObservable
ToEnumerable
AsQ
uery
ab
le
AsEn
um
era
ble
AsQ
bserv
ab
le
AsO
bserv
ab
le
Pull(interactive)
Push(reactive)
Concurre
ncy
(ISch
eduler)
LINQ to *.*
What?
Where
?
How
?
Worker pools Message loopsThreads Distributed
Duality
Hom
o-
icon
ic
IQbservable<T>LINQ to WMI Events
ToQbservableIQueryable<T>
LINQ to SQL
IEnumerable<T>LINQ to Objects
IObservable<T>LINQ to Events
![Page 23: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/23.jpg)
IQbservable<T> - The Dual of IQueryable<T>public interface IQbservable<out T> : IObservable<T>{ Expression Expression { get; } Type ElementType { get; } IQbservableProvider Provider { get; }}
public interface IQbservableProvider{ IQbservable<R> CreateQuery<R>(Expression expression);}
Extended role for some operators
No Execute method
Homoiconic naming
IQbservableIObservable
![Page 24: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/24.jpg)
demo
Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team
LINQ to Twitter
![Page 25: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/25.jpg)
Baked in notion of “where”?
foreach (var p in res.RemoteOn(new SqlScheduler(“server”))) // Process product
Decoupled “what” from “where”
Entry-point for the schema
Custom schedulers could be very rich (e.g. server farm)
The IScheduler abstractionWhere things happen
var ctx = new NorthwindDataContext(); var res = from product in ctx.Products where product.Price > 100m select product.Name;
![Page 26: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/26.jpg)
Expression Tree Remoting
Rx .NET
RxJSstocks.Where(function (t) { return t.Symbol == “MSFT”; }).Select(function (t) { return t.Quote; })
JSON serializer
Observable data source
Retargeting to AJAX
from ticker in stockswhere ticker.Symbol == “MSFT”select ticker.Quote
![Page 27: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/27.jpg)
Query Server
Alice F9F10
chat
talk
var chat = svc.GetStream<Message>(“chat”);
var talk = from msg in chat where msg.From == “Bob” select Translate(msg.Text, “fr”);
talk.Subscribe(s => ShowMessage(s)); F5
chatchatchat
Where
Select
=
msg “Bob”
From
msg
Text
Translate
Translate
chat
Where
Select
=
msg “Bob”
From
msg
Text
![Page 28: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/28.jpg)
Query Server
Alice
chat
talk
var chat = svc.GetStream<Message>(“chat”);
var talk = from msg in chat where msg.From == “Bob” select Translate(msg.Text, “fr”);
talk.Subscribe(s => ShowMessage(s));
Bob F9
chat
F10
var chat = svc.GetStream(“chat”);
Rx.Observable.FromDOMEvent(txt, “Changed”) .Select(function (e) { return txt.Text; }) .Throttle(.5 /* 500 ms */) .Subscribe(function (s) { chat.OnNext(new Message(“Bob”, s)); }); F5
chat
Where
Select
=
msg “Bob”
From
msg
Text
Translate
chat
![Page 29: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/29.jpg)
chat
Query Server
Alice
chat
talk
Bob
chat
Hello Alice! F5
var chat = svc.GetStream(“chat”);
Rx.Observable.FromDOMEvent(txt, “Changed”) .Select(function (e) { return txt.Text; }) .Throttle(.5 /* 500 ms */) .Subscribe(function (s) { chat.OnNext(new Message(“Bob”, s)); });
Bob says:
Hello Alice!
Hello Alice!
Bonjour Alice!
var chat = svc.GetStream<Message>(“chat”);
var talk = from msg in chat where msg.From == “Bob” select Translate(msg.Text, “fr”);
talk.Subscribe(s => ShowMessage(s)); F5
Bob says:
Bonjour Alice!
Where
Select
=
msg “Bob”
From
msg
Text
Translate
“Bonjour Alice!”
From: “Bob”Text: “Hello Alice!”
From: “Bob”Text: “Hello Alice!”
Translate“Bonjour Alice!”
“Hello Alice!”
![Page 30: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/30.jpg)
demo
Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team
Expression Tree Serialization
![Page 31: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/31.jpg)
announcing
Now with support for hosting expression tree based queries (enumerable and observable)
StreamInsight v2.1
![Page 32: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/32.jpg)
LINQ to the UnexpectedModel[ Decisions[Reals, SA, VZ], Goals[ Minimize[20 * SA + 15 * VZ] ], Constraints[ C1 -> 0.3 * SA + 0.4 * VZ >= 2000, C2 -> 0.4 * SA + 0.2 * VZ >= 1500, C3 -> 0.2 * SA + 0.3 * VZ >= 500, C4 -> SA <= 9000, C5 -> VZ <= 6000, C6 -> SA >= 0, C7 -> VZ >= 0 ]]
from m in ctx.CreateModel(new { vz = default(double), sa = default(double)})where 0.3 * m.sa + 0.4 * m.vz >= 2000 && 0.4 * m.sa + 0.2 * m.vz >= 1500 && 0.2 * m.sa + 0.3 * m.vz >= 500where 0 <= m.sa && m.sa <= 9000 && 0 <= m.vz && m.vz <= 6000orderby 20 * m.sa + 15 * m.vzselect m
To compute call Solve
Symbolic, non-persisted
Cost / barrel
Max# barrels
Min# barrels
Refin
em
en
t %
![Page 33: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/33.jpg)
LINQ to the Unexpected
from costSA in GetPriceMonitor("SA")from costVZ in GetPriceMonitor("VZ")
from m in ctx.CreateModel(new { vz = default(double), sa = default(double) })
where 0.3 * m.sa + 0.4 * m.vz >= 2000 && 0.4 * m.sa + 0.2 * m.vz >= 1500 && 0.2 * m.sa + 0.3 * m.vz >= 500where 0 <= m.sa && m.sa <= 9000 && 0 <= m.vz && m.vz <= 6000
orderby costSA * m.sa + costVZ * m.vzselect m
Observabledata sources
Sele
ctM
an
y
Parametersto decision
Subscribe here!
![Page 34: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/34.jpg)
demo
Bart J.F. De SmetSenior Software Development EngineerCloud Programmability Team
LINQ to Constraints
![Page 35: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/35.jpg)
Related Content
DEV413 – Curing your Event Processing Blues using Reactive Extensions (Rx)
Find Me Later The Week At The Ask The Experts
![Page 36: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/36.jpg)
Resources
Connect. Share. Discuss.
http://europe.msteched.com
Learning
Microsoft Certification & Training Resources
www.microsoft.com/learning
TechNet
Resources for IT Professionals
http://microsoft.com/technet
Resources for Developers
http://microsoft.com/msdn
![Page 37: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/37.jpg)
Evaluations
http://europe.msteched.com/sessions
Submit your evals online
![Page 38: LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e575503460f94b50185/html5/thumbnails/38.jpg)
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to
be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS
PRESENTATION.