Scala Days Highlights | BoldRadius
-
Upload
boldradius-solutions -
Category
Documents
-
view
450 -
download
1
Transcript of Scala Days Highlights | BoldRadius
ScalaDays Highlights2015
Who?
● BoldRadius Solutions○ boldradius.com○ Typesafe Partner○ Scala, Akka and Play specialists○ Ottawa, Saskatoon, San Francisco, Boston, Chicago, Montreal, New
York, Toronto
● Michael Nash, VP Capabilities● Adam Murray, VP Business
Our Team at ScalaDays
What?
What was ScalaDays all about?● Held in San Francisco March 16th thru 18th
○ About 800 attendees○ Three categories (intermediate, beginner,
advanced), ○ Four tracks, 55 presentations, three keynotes
● Followed by Two Intensive Training days○ All the Typesafe courses offered
Venue: Fort Mason
The main hall...
Keynotes● Martin Odersky, Chief Architect & Co-Founder at Typesafe
○ Scala - where it came from, where it’s going● Danese Cooper, Distinguished Member of Technical Staff -
Open Source at Paypal○ Open Languages
● Dianne March, Director of Engineering Tools at Netflix
○ Technical Leadership from wherever you are
Quote from James Gosling...
HighlightsToo many great sessions to summarize them all, we have to extract a few recurring themes…
○ Distributed Application Design and Development○ Big Data/Fast Data○ Types and Safe Scala○ Performance and Scalability
Distributed Applications
● Life Beyond the Illusion of Present● Reactive Reference Architectures● Akka in Production: Why and How● Easy Scalability with Akka● Scalable task distribution with Scala, Akka
and Mesos● A Guided Tour of a Distributed Application
Performance and Scalability
● Scala Collections and Performance● Shattering Hadoop’s Large-Scala Sort Record with
Spark and Scala● Type-safe off-heap memory for Scala● Akka in Production: Why and How● Easy Scalability with Akka● The JVM Backend and Optimizer in Scala 2.12
Big Data, Fast Data
● Shattering Hadoop’s Large Scala Sort Record with Spark and Scala
● Scala - The Real Spark of Data Science● Apache Spark: A Large Community Project in Scala● S3 at Scala: Async Scala Client with Play Iteratees and
Composable Operations● The Unreasonable Effectiveness of Scala for Big Data● Scalable task distribution with scala, Akka and Mesos
Types and Safer Scala● Keynote: Scala - where it came from, where it’s going● Towards a Safer Scala● Leveraging Scala Macros for Better Validation● Type-level programming in Scala 101● Improving Correctness with Types● Happy Paths: Functional Constructs in the Wild● The Scalactic Way● Delimited dependency-typed monadic checked
exceptions
The Rest...Many excellent talks were outside these categories● Why Scala.js● Reactive Slick for Database Programming● Exercise in machine learning● Functional Natural Language Processing● If I Only Had a Brain...in Scala● Akka HTTP: the Reactive Web Toolkit● many many others
Highlights of Specific Sessions
A quick taster of what was in some of the more popular sessions...
Scala - Where it came from, where it’s going
Where it’s from, where it’s going“Scala is a gateway drug to Haskell” (in actual fact it’s going well beyond Haskell.)
Slides: http://www.slideshare.net/Odersky/scala-days-san-francisco-45917092
Came from a practical combination of OOP and functional programming- Funny story about hipster syntax (..) instead of [..], <..> instead of [..], ??? instead of <..>- Trend in Type Systems- Scala JS is no longer experimental- TASTY: new scala-specific platform- Introduction to DOT- Type Parameters- Better treatment of effects with implicit Functions instead of Monads
TASTYScala faces challenges:
● binary compatibility● having to pick a platform: JDK( 7,8,9,10, ?) or Javascript.
Proposing a scala-specific platform called TASTY, (serialized typed abstract syntax tree),as an intermediate representation before bytecode - carries metadata of types, can be compiled with different versions of JDK, and to Javascript.
Tasty will enable:● instrumentation● optimization● code analysis● refactoring● publish once run anywhere● automated remapping to solve binary compatibility issues.
Explorations:Implicits that compose
Explorations:Hope to find something cooler than Monads to handle effects.● Monads don’t commute● Require Monad transformers for composition● Monad transformers make Martin’s head explode
Toward a Safer ScalaLeif Wickland
http://tinyurl.com/sd15lint
● Scalac enables some error-prone code.○ head of empty List?
● Using Static Analysis to detect errors early● IDE based solutions
○ Inconsistencies○ If not in release build process, doesn’t exist
● Web-based analysis○ outside of compile loop○ relatively immature analysis
Scalac switches
● Scala Style● Wart Remover● Linter● Watch
○ Abide○ Scapegoat○ Supersafe
Life Beyond the Illusion of PresentJonas Bonér:
The idea of the present is an illusion. Everything we see, hear and feel is just an echo from the past. But this illusion has influenced us and the way we view the world in so many ways.
There is no present, all we have is facts derived from the merging of multiple pasts. The truth is closer to Einstein’s physics where everything is relative to one’s perspective. As developers we need to wake up and break free from the perceived reality of living in a single globally consistent present.
The advent of multicore and cloud computing architectures meant that most applications today are distributed systems—multiple cores separated by the memory bus or multiple nodes separated by the network—which puts a harsh end to this illusion.
The only way to design truly scalable and performant systems that can construct a sufficiently consistent view of history—and thereby our local “present”—is by treating time as a first class construct in our programming model and to model the present as facts derived from the merging of multiple concurrent pasts.
How do we deal with failure and communication unreliability in real life?Confirmation and repetition
We can’t force the world into a globally consistent present (CRUD).
Mentioned 2 paradigms/theories: ● CALM (consistency as logical monotonicity)● CRDT (Commutative Replicated Data Type)
CRDT (Commutative Replicated Data Type) eventually consistent data types● minimize contention / coordination in a distributed system.● set, maps, graphs: rich data types.● monotonic merge function: all state change is monotonically increasing, no way
back.
Improving Correctness with Typeshttp://workday.github.io
● Using Wrapper Types (aka Tiny Types) instead of primitives● Never use null, or throw Exceptions● use === org.scalactic.TypeCheckedTripleEquals
○ requires the types of the two values compared to be in a subtype/supertype● Never Use Non-Empty Lists when a list must be populated (org.scalactic.Every)● Use Type Tags (ala Shapeless, Scalaz)● Use Path Dependent Types● Other reading
○ Self recursive types○ Phantom Types○ Shapeless○ Scalactic
Types: Defensive Programming. Fail Fast. Design By Contract.
Function Passing Style Heather Miller:
● A new programming model called function passing designed to overcome many of imperative / weakly-typed issues found in traditional “big data” processing systems.
● Provides a more principled substrate on which to build data-centric distributed systems.
● Pass safe, well-typed serializable functions to immutable distributed data● Based on her work on Pickling and Spores● Uses Spores (Serializable Functions) for a distributed model.● Kind of an Inverse of the Actor Model● Stateless. data is stationary, functions are passed around.● Uses Data Silos accessed through a Silo Ref.
The Unreasonable Effectiveness of Scala for Big Data
Dean Wampler● How Hadoop Works - Map Reduce
○ Problems■ Hard to implement algorithms■ The Hadoop API is horrible
● Scalding○ An improved Hadoop API in Scala○ Problems
■ Still uses a batch mode● Spark
○ An elegant, functional API○ Still in Batch Mode, but with mini batches which
approach real time.
Akka HTTP: The Reactive Web ToolkitRoland Kuhn
● Replaces Spray● Uses Akka Streams
○ Sources emit values to the stream○ Sinks receive values, act on them○ Sources can compose using Zip and Graph shapes
● “The pinball interpreter”○ produce data○ move downstream through transformations○ get to the effect○ go up and ask for more data○ Filters interrupt the flow before getting to the effect, make the pinball go back
upstream.● A live coded demonstration of using Streams and Http● Expected timeline for Streams - 4 weeks● Expected timeline for Http - 8 weeks
The Scalactic WayBill Vennershttp://www.slideshare.net/bvenners/the-scalactic-way
ScalaTest: quality through tests
Scalactic: quality through types
SuperSafe: quality through static analysis
Reactive Slick for Database ProgrammingStefan Zeigerhttp://slick.typesafe.com/talks/scaladays2015sf/Reactive_Slick_for_Database_Programming.pdf
Slick 3.0
● JDBC is inherently blocking (and blocking ties up threads) ● Traditional Model
○ Fully synchronous○ One thread per web request ○ Contention for Connections (getConnection blocks)○ Database back-pressure creates more blocked threads○ Doesn’t scale
New Slick Architecture makes use of a new DataType to provide Asynchronous Database I/O
● based on State, IO and Free Monads.● Returns a Future[R]● Creates a separate ExecutionContext, avoiding blocking of current thread● Works with akka-streams to create back pressure, so DB only gives as much data as client can
process.● For performance purposes it pre-fetches some data to keep client busy while it waits for the next
portion from DB.
sealed trait DBIOAction[+R, +S <: NoStream, -E <: Effect]{
def map[R2](f: R => R2)(implicit executor: ExecutionContext): DBIOAction[R2, NoStream, E]
def flatMap[R2, S2 <: NoStream, E2 <: Effect](f: R => DBIOAction[R2, S2, E2])(implicit
executor: ExecutionContext): DBIOAction[R2, S2, E with E2]
...
}
Easy Scalability with Akka
Easy Scalability with Akka
Michael Nash● Reviewed Akka, CQRS, ES● Introduced Distributed DDD● Identical clustered system with DDDD and Without● Gatling performance tests on both
ConductR● Application manager that empowers ops to
deploy distributed systems● Uses Akka, Play, Aka Streams, Akka
Cluster, FSM, Akka Data Replication● How can we run cluster based apps ensuring
the seed nodes are started first?○ State replicated using Data
Replication● How can we consolidate logging?
○ Using Akka Streams● How can we avoid batching?
○ Use Event Driven Architecture● How can monitor/test multiple nodes?
○ Use the visualizer built into conductor● How can we share state among the nodes?
○ Use Akka Data Replication
Scala.js
Building a data-driven map of the world in 10 minutes….
Whats wrong with Webapps?● Everything String/Map[String, String]● Compiler cannot help you!
○ document.getElementByld("my-id")● Javascript!
○ ["10", "10", "10", "10"].map(parseInt) [10, NaN, 2, 3] // WTF
ScalaJS● lets you write webapp in one language● Swap String-typing for Strong-typing● Type Everything!!
○ val res: Future[Seq[String]] = Ajax[Api].list(inputBox.value).call()● IS NOT JAVASCRIPT
○ scala.js> List("10", "10", "10", "10").map(parseInt) List(10, 10, 10, 10) // Yay!
The BoldRadius Crew Packing up...
The End….
ScalaDays Amsterdam!
● June 8th to 10th● Followed by Training
Q & [email protected]@boldradius.com