Parallel-Ready Java Code: Managing Mutation in an Imperative Language
212
Parallel-Ready Java Code Managing Mutation in an Imperative Language JavaDay Kyiv Nov 2015 #parallelready @mauricenaftalin
-
Upload
maurice-naftalin -
Category
Software
-
view
269 -
download
0
Transcript of Parallel-Ready Java Code: Managing Mutation in an Imperative Language
Parallel-Ready Java CodeManaging Mutation in an Imperative Language
JavaDay KyivNov 2015
#parallelready@mauricenaftalin
#parallelready @mauricenaftalin
Maurice Naftalin
Developer, designer, architect, teacher, learner, writer
#parallelready @mauricenaftalin
Maurice Naftalin
Developer, designer, architect, teacher, learner, writer
JavaOne Rock Star
Java Champion
#parallelready @mauricenaftalin
Repeat offender:
Maurice Naftalin
#parallelready @mauricenaftalin
Repeat offender:
Maurice NaftalinJava 5
#parallelready @mauricenaftalin
Repeat offender:
Maurice NaftalinJava 5 Java 8
#parallelready @mauricenaftalin
Ideas in the Stream API
– Pipes/Filters
– Parallelism
• Recursive decomposition
• Stream-transforming (aka intermediate) operations
• Parallel merging
#parallelready @mauricenaftalin
Pipes and Filters
• Venerable Unix tool-building pattern:ps -ef | grep login | cut -c 50- | head
• and in Enterprise Integration Patterns
#parallelready @mauricenaftalin
Advantages of this pattern
Pipes and Filters
#parallelready @mauricenaftalin
ps -ef | grep login | cut -c 50- | head
Advantages of this pattern
Pipes and Filters
#parallelready @mauricenaftalin
- no intermediate variables
ps -ef | grep login | cut -c 50- | head
Advantages of this pattern
Pipes and Filters
#parallelready @mauricenaftalin
- no intermediate variables
- less (or no) intermediate storage
ps -ef | grep login | cut -c 50- | head
Advantages of this pattern
Pipes and Filters
#parallelready @mauricenaftalin
- no intermediate variables
- less (or no) intermediate storage
- lazy evaluation
ps -ef | grep login | cut -c 50- | head
Advantages of this pattern
Pipes and Filters
#parallelready @mauricenaftalin
- no intermediate variables
- less (or no) intermediate storage
- lazy evaluation
- flexible tool-building:
ps -ef | grep login | cut -c 50- | head
Advantages of this pattern
Pipes and Filters
#parallelready @mauricenaftalin
Pipes and Filters
Doug McIlroy, inventor of the Unix pipe:
#parallelready @mauricenaftalin
Pipes and Filters
Write programs that do one thing well.
Doug McIlroy, inventor of the Unix pipe:
#parallelready @mauricenaftalin
Pipes and Filters
Write programs that do one thing well.Write programs to work together.
Doug McIlroy, inventor of the Unix pipe:
#parallelready @mauricenaftalin
Pipes and Filters
Write programs that do one thing well.Write programs to work together.
Write programs to handle text streams
Doug McIlroy, inventor of the Unix pipe:
#parallelready @mauricenaftalin
Pipes and Filters
Write programs that do one thing well.Write programs to work together.
Write programs to handle text streams
Doug McIlroy, inventor of the Unix pipe:
XXXXXXXX
#parallelready @mauricenaftalin
Pipes and Filters
Write programs that do one thing well.Write programs to work together.
Write programs to handle text streams
Doug McIlroy, inventor of the Unix pipe:
XXXXXXXX
value objects and primitives
#parallelready @mauricenaftalin
Example Domain
city: City name: Stringage: int
Person
Person amy = new Person(Athens, "Amy", 21); ...
List<Person> people = Arrays.asList(jon, amy, bill);
#parallelready @mauricenaftalin
Pipes and Filters in Java
boolean allAdults = people.stream() .mapToInt(Person::getAge) .allMatch(a -> a >= 21);
Is everyone an adult?
#parallelready @mauricenaftalin
Pipes and Filters in Java
boolean allAdults = people.stream() .mapToInt(Person::getAge) .allMatch(a -> a >= 21);
people.stream() .map(Person::getCity) .forEach(System.out::println);
Is everyone an adult?
Print the names ofthe inhabited cities
Streams
Sequence of values
• Not a collection — may be partially evaluated or exhausted
• Like an iterator, yielding elements for processing
• Not like an iterator, not associated with any storage mechanism
• Sources: collections, arrays, generators, IO channels
• Can be
- parallel
- infinite
• Primitive specialisations: IntStream, LongStream, DoubleStream
#parallelready @mauricenaftalin
Sources – sequential streams
Stream Sources: – supply elements one at a time– implementations of java.util.Spliterator – 40+ methods in platform classes
– including Collection.stream
boolean allAdults = people.stream() .mapToInt(Person::getAge) .allMatch(a -> a >= 21);
#parallelready @mauricenaftalin
Intermediate Operations
Filtering filter()
Mapping map()
One-to-Many Mapping flatMap()
Debugging peek()
Sorting/Deduplicating sorted(), distinct()
Truncating skip(), limit()
#parallelready @mauricenaftalin
filter(s -> s.length() < 4)
Stream<String>
Predicate<String
Stream<String>
filter()
#parallelready @mauricenaftalin
filter(s -> s.length() < 4)
Stream<String>
Predicate<String
Stream<String>
filter()
“jim”
✔
#parallelready @mauricenaftalin
filter(s -> s.length() < 4)
Stream<String>
Predicate<String
Stream<String>
filter()
✔
#parallelready @mauricenaftalin
✖
filter(s -> s.length() < 4)
Stream<String>
Predicate<String
Stream<String>
filter()
✔
#parallelready @mauricenaftalin
✖
filter(s -> s.length() < 4)
Stream<String>
Predicate<String
Stream<String>
filter()
✔
“amy”
#parallelready @mauricenaftalin
✖
filter(s -> s.length() < 4)
Stream<String>
Predicate<String
Stream<String>
filter()
✔
#parallelready @mauricenaftalin
map()
map(Person::getCity)
Stream<Person> Stream<City>
Function<Person,City
#parallelready @mauricenaftalin
map()
map(Person::getCity)
Stream<Person> Stream<City>
Function<Person,City
bill
#parallelready @mauricenaftalin
map()
map(Person::getCity)
Stream<Person> Stream<City>
Function<Person,City
#parallelready @mauricenaftalin
map()
map(Person::getCity)
Stream<Person> Stream<City>
Function<Person,City
amy
#parallelready @mauricenaftalin
map()
map(Person::getCity)
Stream<Person> Stream<City>
Function<Person,City
#parallelready @mauricenaftalin
peek()
peek(System.out::println)
Consumer<T>
System.out::println
T
Stream<T>
#parallelready @mauricenaftalin
flatMapToInt()
Stream<String>
Function<String,IntStream>
IntStream
flatMapToInt(String::chars)
#parallelready @mauricenaftalin
flatMapToInt()
Stream<String>
Function<String,IntStream>
IntStream
'a' 'm' 'y'
flatMapToInt(String::chars)
#parallelready @mauricenaftalin
flatMapToInt()
Stream<String>
Function<String,IntStream>
IntStream
'm' 'y'
flatMapToInt(String::chars)
#parallelready @mauricenaftalin
flatMapToInt()
Stream<String>
Function<String,IntStream>
IntStream
'y'
flatMapToInt(String::chars)
#parallelready @mauricenaftalin
flatMapToInt()
Stream<String>
Function<String,IntStream>
IntStream
flatMapToInt(String::chars)
Ending Streams
Search findFirst(),findAny()anyMatch(), allMatch()
Side-effecting forEach(), forEachOrdered()
Reductions ...
Stream Sinks (terminal operations):
#parallelready @mauricenaftalin
Parallelism
“The best way to write parallel applications is not to have to think about parallelism.”
Guy Steele
#parallelready @mauricenaftalin
Automated Resource Management
#parallelready @mauricenaftalin
Automated Resource Management
#parallelready @mauricenaftalin
Assemblers
Automated Resource Management
#parallelready @mauricenaftalin
AssemblersLinkers
Automated Resource Management
#parallelready @mauricenaftalin
AssemblersLinkersRegister allocation
Automated Resource Management
#parallelready @mauricenaftalin
AssemblersLinkersRegister allocationHeap management
Automated Resource Management
#parallelready @mauricenaftalin
AssemblersLinkersRegister allocationHeap managementVirtual memory
Automated Resource Management
#parallelready @mauricenaftalin
AssemblersLinkersRegister allocationHeap managementVirtual memoryParallelisation
Automated Resource Management
#parallelready @mauricenaftalin
Retooling
#parallelready @mauricenaftalin
Some old habits won’t work any more
Retooling
#parallelready @mauricenaftalin
Some old habits won’t work any moreminimising operation count (eg by reusing previously computed results)
Retooling
#parallelready @mauricenaftalin
Some old habits won’t work any moreminimising operation count (eg by reusing previously computed results)
- be prepared to “waste” operations to reduce communication
Retooling
#parallelready @mauricenaftalin
Some old habits won’t work any moreminimising operation count (eg by reusing previously computed results)
- be prepared to “waste” operations to reduce communication
minimising space usage (by sharing data between operations)
Retooling
#parallelready @mauricenaftalin
Some old habits won’t work any moreminimising operation count (eg by reusing previously computed results)
- be prepared to “waste” operations to reduce communication
minimising space usage (by sharing data between operations)
- parallel programs trade space for reduced coupling
Retooling
#parallelready @mauricenaftalin
Some old habits won’t work any moreminimising operation count (eg by reusing previously computed results)
- be prepared to “waste” operations to reduce communication
minimising space usage (by sharing data between operations)
- parallel programs trade space for reduced coupling
linear decomposition
Retooling
#parallelready @mauricenaftalin
Some old habits won’t work any moreminimising operation count (eg by reusing previously computed results)
- be prepared to “waste” operations to reduce communication
minimising space usage (by sharing data between operations)
- parallel programs trade space for reduced coupling
linear decomposition
- processing one thing at a time to accumulate results
Retooling
#parallelready @mauricenaftalin
Some old habits won’t work any moreminimising operation count (eg by reusing previously computed results)
- be prepared to “waste” operations to reduce communication
minimising space usage (by sharing data between operations)
- parallel programs trade space for reduced coupling
linear decomposition
- processing one thing at a time to accumulate results
Retooling
No more accumulators!
#parallelready @mauricenaftalin
Seriously, No More Accumulators??
#parallelready @mauricenaftalin
Seriously, No More Accumulators??
No more accumulators!
#parallelready @mauricenaftalin
Seriously, No More Accumulators??
Divide And Conquer algorithms instead
- symmetric merge operations
No more accumulators!
#parallelready @mauricenaftalin
Seriously, No More Accumulators??
Divide And Conquer algorithms instead
- symmetric merge operationsMajor impact on terminal operations
- almost all are accumulations of some kind
No more accumulators!
#parallelready @mauricenaftalin
• Using an accumulator:
Simple Example – Summation
int[] vals = new int[100];Arrays.setAll(vals, i -> i);
int sum = 0;for (int i = 0 ; i < vals.length ; i++) { sum += vals[i];}
0 1 2 3
+
+
+
#parallelready @mauricenaftalin
• Avoiding an accumulator:
Simple Example – Summation
#parallelready @mauricenaftalin
• Avoiding an accumulator:
Simple Example – Summation
#parallelready @mauricenaftalin
+
+
+
• Avoiding an accumulator:
Simple Example – Summation
int[] vals = new int[100];Arrays.setAll(vals, i -> i);
OptionalInt sum = Arrays.stream(vals) .reduce((a,b) -> a + b);
1 2 30
#parallelready @mauricenaftalin
+ +
+
• Avoiding an accumulator:
Simple Example – Summation
int[] vals = new int[100];Arrays.setAll(vals, i -> i);
OptionalInt sum = Arrays.stream(vals) .reduce((a,b) -> a + b);
1 2 30
#parallelready @mauricenaftalin
+ +
+
• Avoiding an accumulator:
Simple Example – Summation
int[] vals = new int[100];Arrays.setAll(vals, i -> i);
OptionalInt sum = Arrays.stream(vals) .reduce((a,b) -> a + b);
BinaryOperator must be associative!
1 2 30
#parallelready @mauricenaftalin
+ +
+
• Avoiding an accumulator:
Simple Example – Summation
int[] vals = new int[100];Arrays.setAll(vals, i -> i);
OptionalInt sum = Arrays.stream(vals) .reduce((a,b) -> a + b);
BinaryOperator must be associative!a + (b + c) = (a + b) + c
1 2 30
#parallelready @mauricenaftalin
Reduction of Primitives0 1 2
+
+
+
+
+
+
+
3 4 5
0 0
#parallelready @mauricenaftalin
Reduction of Primitives0 1 2
+
+
+
+
+
+
+
3 4 5
Identity
BinaryOperator
0 0
#parallelready @mauricenaftalin
Reduction of Primitives0 1 2
+
+
+
+
+
+
+
3 4 5
Identity
BinaryOperator
0 0
#parallelready @mauricenaftalin
Reduction of Mutable Objects?
add
add
add
add
add
add
combine
[ ] [ ]
Identity
Accumulator
Combiner
e0 e1 e2 e3 e4 e5
#parallelready @mauricenaftalin
Reduction of Mutable Objects?
add
add
add
add
add
add
combine
[ ] [ ]
Identity
Accumulator
Combiner
e0 e1 e2 e3 e4 e5
Not�Parallel-ReadyNot�Parallel-Ready
#parallelready @mauricenaftalin
Collector
combine
()->[]
Supplier
Accumulator
Combiner
e0 e1 e2 e3 e4 e5
()->[]
add
add
add
add
add
add
#parallelready @mauricenaftalin
Using the Predefined Collectors
Predefined Standalone Collectors – from factory methods in Collectors class
• toList(), toSet(), toMap(), joining()
• toMap(), toCollection()
• groupingBy(), partitioningBy()
#parallelready @mauricenaftalin
Using the Predefined Collectors
Predefined Standalone Collectors – from factory methods in Collectors class
• toList(), toSet(), toMap(), joining()
• toMap(), toCollection()
• groupingBy(), partitioningBy()
framework providesthe Supplier
#parallelready @mauricenaftalin
Using the Predefined Collectors
Predefined Standalone Collectors – from factory methods in Collectors class
• toList(), toSet(), toMap(), joining()
• toMap(), toCollection()
• groupingBy(), partitioningBy()
user providesthe Supplier
framework providesthe Supplier
#parallelready @mauricenaftalin
Using the Predefined Collectors
Predefined Standalone Collectors – from factory methods in Collectors class
• toList(), toSet(), toMap(), joining()
• toMap(), toCollection()
• groupingBy(), partitioningBy()
user providesthe Supplier
framework providesthe Supplier
produce a classification map
#parallelready @mauricenaftalin
Simple Collector – toSet()people.stream().collect(Collectors.toSet())
#parallelready @mauricenaftalin
Collector<Person,?,Set<Pers
Stream<Person> Set<Person>
Simple Collector – toSet()people.stream().collect(Collectors.toSet())
#parallelready @mauricenaftalin
Collector<Person,?,Set<Pers
Stream<Person> Set<Person>bill
Simple Collector – toSet()people.stream().collect(Collectors.toSet())
#parallelready @mauricenaftalin
Collector<Person,?,Set<Pers
Stream<Person> Set<Person>bill
Simple Collector – toSet()people.stream().collect(Collectors.toSet())
#parallelready @mauricenaftalin
Collector<Person,?,Set<Pers
Stream<Person> Set<Person>bill
Simple Collector – toSet()people.stream().collect(Collectors.toSet())
#parallelready @mauricenaftalin
Collector<Person,?,Set<Pers
Stream<Person> Set<Person>billjon
Simple Collector – toSet()people.stream().collect(Collectors.toSet())
#parallelready @mauricenaftalin
Collector<Person,?,Set<Pers
Stream<Person> Set<Person>billjon
Simple Collector – toSet()people.stream().collect(Collectors.toSet())
#parallelready @mauricenaftalin
Collector<Person,?,Set<Pers
Stream<Person> Set<Person>amy
billjon
Simple Collector – toSet()people.stream().collect(Collectors.toSet())
#parallelready @mauricenaftalin
Collector<Person,?,Set<Pers
Stream<Person> Set<Person>
amy
billjon
Simple Collector – toSet()people.stream().collect(Collectors.toSet())
#parallelready @mauricenaftalin
Collector<Person,?,Set<Pers
Set<Person>
amy
billjon
Simple Collector – toSet()people.stream().collect(Collectors.toSet())
#parallelready @mauricenaftalin
Collector<Person,?,Set<Pers
Set<Person>{ , , }amybilljon
Simple Collector – toSet()people.stream().collect(Collectors.toSet())
#parallelready @mauricenaftalin
toMap(Function<T,K> keyMapper, Function<T,U> valueMapper)
people.stream().collect( Collectors.toMap(
Person::getCity, Person::getName))
#parallelready @mauricenaftalin
toMap(Function<T,K> keyMapper, Function<T,U> valueMapper)
Stream<Person>
toMap() Map<City,String>
Collector<Person,?,Map<City,String>
#parallelready @mauricenaftalin
toMap(Function<T,K> keyMapper, Function<T,U> valueMapper)
Stream<Person>
toMap() Map<City,String>
bill
#parallelready @mauricenaftalin
toMap(Function<T,K> keyMapper, Function<T,U> valueMapper)
Stream<Person>
toMap() Map<City,String>
bill
Person::getCity
#parallelready @mauricenaftalin
toMap(Function<T,K> keyMapper, Function<T,U> valueMapper)
Stream<Person>
toMap() Map<City,String>London
bill
#parallelready @mauricenaftalin
toMap(Function<T,K> keyMapper, Function<T,U> valueMapper)
Stream<Person>
toMap() Map<City,String>London
bill Person::getName
#parallelready @mauricenaftalin
toMap(Function<T,K> keyMapper, Function<T,U> valueMapper)
Stream<Person>
toMap() Map<City,String>London “Bill”
bill Person::getName
#parallelready @mauricenaftalin
toMap(Function<T,K> keyMapper, Function<T,U> valueMapper)
Stream<Person>
toMap() Map<City,String>London “Bill”
#parallelready @mauricenaftalin
toMap(Function<T,K> keyMapper, Function<T,U> valueMapper)
Stream<Person>
toMap() Map<City,String>
amy
London “Bill”
#parallelready @mauricenaftalin
toMap(Function<T,K> keyMapper, Function<T,U> valueMapper)
Stream<Person>
toMap() Map<City,String>
Athensamy
London “Bill”
#parallelready @mauricenaftalin
toMap(Function<T,K> keyMapper, Function<T,U> valueMapper)
Stream<Person>
toMap() Map<City,String>
“Amy”Athens
London “Bill”
#parallelready @mauricenaftalin
toMap(Function<T,K> keyMapper, Function<T,U> valueMapper)
Stream<Person>
toMap() Map<City,String>
“Amy”Athensjon
London “Bill”
#parallelready @mauricenaftalin
toMap(Function<T,K> keyMapper, Function<T,U> valueMapper)
Stream<Person>
toMap() Map<City,String>
“Amy”Athens
London “Bill”
Tulsa “Jon”
#parallelready @mauricenaftalin
toMap(Function<T,K> keyMapper, Function<T,U> valueMapper)
toMap() Map<City,String>
“Amy”Athens
London “Bill”
Tulsa “Jon”
#parallelready @mauricenaftalin
toMap(Function<T,K> keyMapper, Function<T,U> valueMapper)
toMap() Map<City,String>
“Amy”AthensLondon “Bill”
Tulsa “Jon”
#parallelready @mauricenaftalin
groupingBy(Function<T,K> classifier)Uses the classifier function to make a classification mapping
Like toMap(), except that the values placed in the map are lists of the elements, one List corresponding to each classification key:
For example, use Person.getCity() to make a Map<City,List<Person>>
Map<City,List<Person>> peopleByCity = people.stream(). collect(Collectors.groupingBy(Person::getCity));
#parallelready @mauricenaftalin
Stream<Person>
groupingBy() Map<City,List<Person>>
bill
jon
amy Athens
London
Collector<Person,?,Map<City,List<Person>>>
groupingBy(Function<Person,City>)
Classifier
Person→City
#parallelready @mauricenaftalin
Stream<Person>
groupingBy() Map<City,List<Person>>
bill
jon
amy Athens
London
groupingBy(Function<Person,City>)
#parallelready @mauricenaftalin
Stream<Person>
groupingBy() Map<City,List<Person>>
bill
jon
amy Athens
bill London
groupingBy(Function<Person,City>)
#parallelready @mauricenaftalin
Stream<Person>
groupingBy() Map<City,List<Person>>
bill
jon
amy Athens
billLondon
groupingBy(Function<Person,City>)
[ ]
#parallelready @mauricenaftalin
Stream<Person>
groupingBy() Map<City,List<Person>>
bill
jon
amy Athens [ ]
bill
amy
London
groupingBy(Function<Person,City>)
[ ]
#parallelready @mauricenaftalin
Stream<Person>
groupingBy() Map<City,List<Person>>
bill
jon
amy Athens [ ]
[ , ]bill
amy
jonLondon
groupingBy(Function<Person,City>)
#parallelready @mauricenaftalin
groupingBy() Map<City,List<Person>>
bill
jon
amy Athens [ ]
[ , ]bill
amy
jonLondon
groupingBy(Function<Person,City>)
#parallelready @mauricenaftalin
groupingBy() Map<City,List<Person>>
bill
jon
amy
Athens [ ]
[ , ]bill
amy
jonLondon
groupingBy(Function<Person,City>)
groupingBy(classifier, downstream)
Map<City,List<Person>> peopleByCity = people.stream(). collect(Collectors.groupingBy(Person::getCity,toSet()));
Uses the classifier function to make a classification mapping into a container defined by a downstream Collector
Like toMap(), except that the values placed in the map are containers of the elements, one container corresponding to each classification key:
For example, use Person.getCity() to make a Map<City,Set<Person>>
Map<City,Set<Person>> peopleByCity = people.stream(). collect(Collectors.groupingBy(Person::getCity,toSet()));
#parallelready @mauricenaftalin
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()Map<City,Set<Person>>
bill
jon
amy
London
Athens
DownstreamCollector
—toSet()
Stream<Person>
Classifier
Person→City
#parallelready @mauricenaftalin
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()Map<City,Set<Person>>
bill
jon
amy
London
Athens
#parallelready @mauricenaftalin
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()Map<City,Set<Person>>
bill
jon
amy
London
Athens
bill
#parallelready @mauricenaftalin
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()Map<City,Set<Person>>
bill
jon
amy
London
Athens
bill
#parallelready @mauricenaftalin
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()Map<City,Set<Person>>
bill
jon
amy
London
Athens
bill
#parallelready @mauricenaftalin
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()Map<City,Set<Person>>
bill
jon
amy
London
Athensamy
bill
#parallelready @mauricenaftalin
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()Map<City,Set<Person>>
bill
jon
amy
London
Athens amy
bill
#parallelready @mauricenaftalin
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()Map<City,Set<Person>>
bill
jon
amy
London
Athens amy
bill
#parallelready @mauricenaftalin
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()Map<City,Set<Person>>
bill
jon
amy
London
Athens amy
jon bill
#parallelready @mauricenaftalin
groupingBy(Function classifier,Collector downstream))
Stream<Person>
groupingBy()Map<City,Set<Person>>
bill
jon
amy
London
Athens amy
jonbill
#parallelready @mauricenaftalin
groupingBy(Function classifier,Collector downstream))
groupingBy()Map<City,Set<Person>>
bill
jon
amy
London
Athens amy
jonbill
#parallelready @mauricenaftalin
groupingBy(Function classifier,Collector downstream))
groupingBy()Map<City,Set<Person>>
bill
jon
amy
London
Athens amy
jonbill
#parallelready @mauricenaftalin
groupingBy(Function classifier,Collector downstream))
groupingBy()Map<City,Set<Person>>
bill
jon
amy
London
Athens { }amy
{ , }jonbill
#parallelready @mauricenaftalin
groupingBy(Function classifier,Collector downstream))
groupingBy()Map<City,Set<Person>>
bill
jon
amy
London
Athens { }amy
{ , }jonbill
#parallelready @mauricenaftalin
Specialised Downstream Collectors
Stream<Person>
groupingBy()
bill
jon
amy
London
Athens
amyjonbillDownstream
CollectorClassifier
Duals of convenience reductionsMapping collector
“Ordinary”collectors +
Concurrent Collection
Concurrent Collection
Concurrent CollectionThread safety is guaranteed by the framework
• Even for non-threadsafe containers!
Concurrent CollectionThread safety is guaranteed by the framework
• Even for non-threadsafe containers!
But at a price... So what if your container is already threadsafe?
• a concurrentMap implementation?
Concurrent CollectionThread safety is guaranteed by the framework
• Even for non-threadsafe containers!
But at a price... So what if your container is already threadsafe?
• a concurrentMap implementation?
Concurrent CollectionThread safety is guaranteed by the framework
• Even for non-threadsafe containers!
But at a price... So what if your container is already threadsafe?
• a concurrentMap implementation?
Every overload of toMap() and groupingBy() has a dual
• toConcurrentMap(...)
• groupingByConcurrent(...)
Writing a Collector
Writing a Collector
Why would you want to?
Writing a Collector
Why would you want to?
• accumulate to a container that doesn’t implement Collection
Writing a Collector
Why would you want to?
• accumulate to a container that doesn’t implement Collection
• share state between values being collected
Writing a Collector
Why would you want to?
• accumulate to a container that doesn’t implement Collection
• share state between values being collected
• to show how to parallelize “iterative-looking” problems
Oprah’s Problem
Oprah’s Problem
How to find a book?
Oprah’s Problem
How to find a book?
336640
144624
Page count 239640
3841104
172400
What Oprah Needs
New Earth 0Poisonwood Bible 336
Night 976A Fine Balance 1120
...
Cumulative page counts
Creating A Partial Solution
,“Night” 144 976 “AFB” 624 1120 ]“NE” 336 0 “PB” 640 336[ , ,
Creating A Partial Solution
,“Night” 144 0[ “AFB” 624 144 ]“NE” 336 0 “PB” 640 336[ ],
,“Night” 144 976 “AFB” 624 1120 ]“NE” 336 0 “PB” 640 336[ , ,
Creating A Partial Solution
combiner
,“Night” 144 0[ “AFB” 624 144 ]“NE” 336 0 “PB” 640 336[ ],
,“Night” 144 976 “AFB” 624 1120 ]“NE” 336 0 “PB” 640 336[ , ,
Creating A Partial Solution
combiner
,“Night” 144 0[ “AFB” 624 144 ]“NE” 336 0 “PB” 640 336[ ],
,“Night” 144 976 “AFB” 624 1120 ]“NE” 336 0 “PB” 640 336[ , ,
Supplier and Accumulator
“NE” 336 0 “PB” 640 336[ ],
]“NE” 336 0[
][
aNewEarth thePoisonwoodBible
accumulator
accumulator
supplier
Combiner
,“Night” 144 976 “AFB” 624 1120 ]“NE” 336 0 “PB” 640 336[ , ,
Combiner
,“Night” 144 0[ “AFB” 624 144 ]“NE” 336 0 “PB” 640 336[ ],
,“Night” 144 976 “AFB” 624 1120 ]“NE” 336 0 “PB” 640 336[ , ,
Combiner
combiner
,“Night” 144 0[ “AFB” 624 144 ]“NE” 336 0 “PB” 640 336[ ],
,“Night” 144 976 “AFB” 624 1120 ]“NE” 336 0 “PB” 640 336[ , ,
Combiner
combiner
,“Night” 144 0[ “AFB” 624 144 ]“NE” 336 0 “PB” 640 336[ ],
,“Night” 144 976 “AFB” 624 1120 ]“NE” 336 0 “PB” 640 336[ , ,
976
Combiner
combiner
,“Night” 144 0[ “AFB” 624 144 ]“NE” 336 0 “PB” 640 336[ ],
,“Night” 144 976 “AFB” 624 1120 ]“NE” 336 0 “PB” 640 336[ , ,
976
Combiner
combiner
,“Night” 144 0[ “AFB” 624 144 ]“NE” 336 0 “PB” 640 336[ ],
,“Night” 144 976 “AFB” 624 1120 ]“NE” 336 0 “PB” 640 336[ , ,
976
#parallelready @mauricenaftalin
Sources – parallel streams
Implemented by a Spliterator
#parallelready @mauricenaftalin
Sources – parallel streams
Implemented by a Spliterator
#parallelready @mauricenaftalin
Sources – parallel streams
Implemented by a Spliterator
#parallelready @mauricenaftalin
Sources – parallel streams
Implemented by a Spliterator
#parallelready @mauricenaftalin
Sources – parallel streams
Implemented by a Spliterator
#parallelready @mauricenaftalin
Sources – parallel streams
Implemented by a Spliterator
#parallelready @mauricenaftalin
Sources – parallel streams
Implemented by a Spliterator
#parallelready @mauricenaftalin
Sources – parallel streams
Implemented by a Spliterator
#parallelready @mauricenaftalin
Sources – parallel streams
Implemented by a Spliterator
#parallelready @mauricenaftalin
x2
x0
x1
x3
x0
x1
x2
x3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
Visualizing Stream Operations
#parallelready @mauricenaftalin
x2
x0
x1
x3
x0
x1
x2x3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
Visualizing Stream Operations
#parallelready @mauricenaftalin
x2
x0
x1
x3
y0
y1
y2
y3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
Visualizing Stream Operations
#parallelready @mauricenaftalin
x2
x0
x1
x3
y0y1y2y3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
Visualizing Stream Operations
#parallelready @mauricenaftalin
x2
x0
x1
x3
x0
x1
x2
x3
Visualizing Stream Operations
Intermediate Op(s)
(Mutable)Reduction
Spliterator
#parallelready @mauricenaftalin
x2
x0
x1
x3
x0
x1
x2
x3
Visualizing Stream Operations
Intermediate Op(s)
(Mutable)Reduction
Spliterator
#parallelready @mauricenaftalin
x2
x0
x1
x3
x0
x1
x2x3
Visualizing Stream Operations
Intermediate Op(s)
(Mutable)Reduction
Spliterator
#parallelready @mauricenaftalin
x2
x0
x1
x3
x0
x1
x2x3
Visualizing Stream Operations
Intermediate Op(s)
(Mutable)Reduction
Spliterator
#parallelready @mauricenaftalin
x2
x0
x1
x3
x0
x1
x2x3
Visualizing Stream Operations
Intermediate Op(s)
(Mutable)Reduction
Spliterator
#parallelready @mauricenaftalin
x2
x0
x1
x3
Visualizing Stream Operations
y0
y1
y2
y3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
#parallelready @mauricenaftalin
x2
x0
x1
x3
Visualizing Stream Operations
y0y1y2y3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
Visualizing Stream Operations
x2
x1
x3
x1
x2
x3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
x0x0
peek(System.out::print)
Visualizing Stream Operations
x2
x1
x3
x1
x2
x3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
x0x0
peek(System.out::print)
Visualizing Stream Operations
x2
x1
x3
x1
x2
x3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
x0
x0
x0
peek(System.out::print)
Visualizing Stream Operations
x2
x1
x3
x1
x2
x3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
x0
x0
x0
peek(System.out::print)
Visualizing Stream Operations
x2
x1
x3
x1
x2x3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
x0
x2
x0
x0
peek(System.out::print)
Visualizing Stream Operations
x2
x1
x3
x1
x2x3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
x0x2
x0
x0
peek(System.out::print)
Visualizing Stream Operations
x2
x1
x3
x1
x2x3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
x0x2
x0
x0
peek(System.out::print)
Visualizing Stream Operations
x2
x1
x3
x1
x2x3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
x0x2
x0
x3
x0
peek(System.out::print)
Visualizing Stream Operations
x2
x1
x3
x1
x2x3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
x0x2
x0
x3
x0
peek(System.out::print)
Visualizing Stream Operations
x2
x1
x3
x1
x2x3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
x0x2
x0
x3
x0
peek(System.out::print)
Visualizing Stream Operations
x2
x1
x3
x1
x2x3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
x0x2
x0
x3
x0
peek(System.out::print)
Visualizing Stream Operations
x2
x1
x3
y0
y1
y2
y3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
x0x2
x0
x3peek(System.out::print)
Visualizing Stream Operations
x2
x1
x3
y0y1y2y3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
x0x2
x0
x3peek(System.out::print)
Visualizing Stream Operations
x2
x1
x3
Intermediate Op(s)
(Mutable)Reduction
Spliterator
x0x2
x0
x3peek(System.out::print)
#parallelready @mauricenaftalin
What Does Parallel-Ready Mean?
#parallelready @mauricenaftalin
What Does Parallel-Ready Mean?
– sequential execution is no longer the default
#parallelready @mauricenaftalin
What Does Parallel-Ready Mean?
– sequential execution is no longer the default– it’s still often more efficient, but that will change
#parallelready @mauricenaftalin
What Does Parallel-Ready Mean?
– sequential execution is no longer the default– it’s still often more efficient, but that will change– code must be agnostic about its execution mode
#parallelready @mauricenaftalin
What Does Parallel-Ready Mean?
– sequential execution is no longer the default– it’s still often more efficient, but that will change– code must be agnostic about its execution mode– operations in the Stream API have equivalent effect
#parallelready @mauricenaftalin
What Does Parallel-Ready Mean?
– sequential execution is no longer the default– it’s still often more efficient, but that will change– code must be agnostic about its execution mode– operations in the Stream API have equivalent effect- sequential or parallel
#parallelready @mauricenaftalin
What Does Parallel-Ready Mean?
– sequential execution is no longer the default– it’s still often more efficient, but that will change– code must be agnostic about its execution mode– operations in the Stream API have equivalent effect- sequential or parallel
- may be equally non-deterministic
#parallelready @mauricenaftalin
What Does Parallel-Ready Mean?
– sequential execution is no longer the default– it’s still often more efficient, but that will change– code must be agnostic about its execution mode– operations in the Stream API have equivalent effect- sequential or parallel
- may be equally non-deterministic
– decouples execution mode from functional behavior
#parallelready @mauricenaftalin
What Does Parallel-Ready Mean?
– sequential execution is no longer the default– it’s still often more efficient, but that will change– code must be agnostic about its execution mode– operations in the Stream API have equivalent effect- sequential or parallel
- may be equally non-deterministic
– decouples execution mode from functional behavior– parallel-ready is not only about parallel execution!
#parallelready @mauricenaftalin
What Does Parallel-Ready Mean?
– sequential execution is no longer the default– it’s still often more efficient, but that will change– code must be agnostic about its execution mode– operations in the Stream API have equivalent effect- sequential or parallel
- may be equally non-deterministic
– decouples execution mode from functional behavior– parallel-ready is not only about parallel execution!- or even mainly, for now
#parallelready @mauricenaftalin
So, Is Your Code Parallel-Ready?
Most unlikely. But it can be!
The Stream API provides a framework and a lot of support for writing PR code. You just need to co-operate with it.
- it depends on you behaving sanely
#parallelready @mauricenaftalin
Using Streams Sanely
Rules in the JavaDoc for java.util.stream Stateless behavior:- avoid stateful behavioral parameters
- “one whose result depends on any state which might change during the execution of the stream pipeline”
...
#parallelready @mauricenaftalin
Using Streams Sanely
Rules in the JavaDoc for java.util.stream Stateless behavior:- avoid stateful behavioral parameters
- “one whose result depends on any state which might change during the execution of the stream pipeline”
...
Safely
#parallelready @mauricenaftalin
Using Streams Sanely
Rules in the JavaDoc for java.util.stream ...Non-interference- don’t modify a stream’s data source during execution
- you may if it’s a thread-safe class
Don’t rely on side-effects- use for debugging only
#parallelready @mauricenaftalin
Using Streams Sanely
Rules in the JavaDoc for java.util.stream ...Non-interference- don’t modify a stream’s data source during execution
- you may if it’s a thread-safe class
Don’t rely on side-effects- use for debugging only
Safely
#parallelready @mauricenaftalin
Conclusion
The Streams API offers a framework for- programming bulk data operations in parallel-ready style
- easy parallellism, with thread-safe access to non-threadsafe containers
All we have to do is co-operate with it!Parallel-Ready code starts today!
#parallelready @mauricenaftalin
Resources
www.lambdafaq.orgwww.masteringlambdas.org
