sbt, history of JSON libraries,

microservices, and schema evolution

Eugene Yokota (@eed3si9n)February, 2017

• Scala hobbyist since 2010

• scalaxb (XML data binding)

• treehugger.scala

• sbt-assembly, sbt-buildinfo, etc

• “learning Scalaz” / “herding Cats”

• ScalaMatsuri

• Lightbend/Typesafe since 2014

• tech lead of Reactive Platform team

• current maintainer / tech lead of sbt

who is this guy (@eed3si9n)?

Lightbend Production Suite

big picture: development at scale

• How do you scale a technological organization?

• Goal: Sustainable development

development at scale

• Bezos mandate (written circa 2002 before AWS) https://plus.google.com/+RipRowan/posts/eVeouesvaVX

microservice is a social stack

1. All teams will henceforth expose their data and functionality through service interfaces.

2. Teams must communicate with each other through these interfaces.

3. There will be no other form of inter-process communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.

4. It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols -- doesn't matter. Bezos doesn't care.

5. All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

6. Anyone who doesn't do this will be fired.

Brief history of

JSON libraries

Eugene Yokota (@eed3si9n)2016

Brief history of JSON libraries

Dispatch JSON

literaljson

Lift JSON

sjsonSpray JSON

Play JSON

Argonaut

json4s

Circe

Jawn

SLIP-28 JSON

Rapture JSON

sjson-new

2009 2010 2014

2008

Two books

Programming in Scala

import scala.util.parsing.combinator._ class JSON extends JavaTokenParsers { def value : Parser[Any] = obj | arr | stringLiteral | floatingPointNumber | "null" | "true" | "false" def obj : Parser[Any] = "{"~repsep(member, ",")~"}" def arr : Parser[Any] = "["~repsep(value, ",")~"]" def member: Parser[Any] = stringLiteral~":"~value }

https://www.artima.com/pins1ed/combinator-parsing.html#31.4

• Chapter 5. Writing a library: working with JSON data

Real World Haskell

data JValue = JString String | JNumber Double | JBool Bool | JNull | JObject [(String, JValue)] | JArray [JValue] deriving (Eq, Ord, Show)

2009

• https://github.com/dispatch/dispatch/commit/41edb939baa5c6edb4378c1bd8e1d2f10f3350f2

• Contributed by Jorge Ortiz

• Parsing using parser combinator

• Values stored in AST, JsValue

Dispatch JSON

• https://github.com/jonifreeman/literaljson

• Authored by Joni Freeman

• Custom parser

• Values stored in AST, JValue

• On August 11, 2009, Joni contributed literaljson to Lift, and became Lift-JSON

jonifreeman/literaljson

2010

• https://github.com/debasishg/sjson

• Authored by Debasish Ghosh

• sjson: Now offers Type Class based JSON Serialization in Scala

• Uses Dispatch JSON for AST

sjson

Typeclass

• Allows adding capability to a class after the fact in a typesafe manner.

Typeclass

trait Eq[A] {   def equals(x: A, y: A): Boolean }

• Eq typeclass can enable === operator to compare only the supported types, and prevent compilation of "1" === 1

• Scala Implicits : Type Classes Here I Come

JsonFormat

trait CanRead[A] {   def reads(json: JsValue): A } trait CanWrite[A] {   def writes(a: A): JsValue } trait JsonFormat[A] extends CanRead[A] with CanWrite[A]

implicit val intFormat: JsonFormat[Int] = ...

JsonFormat• JsonFormat typeclass allows conversion to and

from an arbitrary type to a JSON AST.

2011

• https://github.com/spray/spray-json

• Authored by Mathias Dönitz

• Original parser

• Ported AST from Dispatch JSON and type classes from sjson

spray-json

• https://github.com/playframework/playframework/commit/63448578b15dcc7bf4806878c7b3aa4c74193af6

• Started out as port of Dispatch JSON and sjson typeclasses, but quickly added its own implementations.

Play JSON

2012

• http://argonaut.io/

• Purely functional JSON library

• Authored by Mark Hibberd, Tony Morris, Sean Parsons

• Uses Scalaz or Cats

• Very feature rich (Lenses, Cursor, History Cursor)

Argonaut

2013

• https://github.com/json4s/json4s

• Fork of Lift-json. JSON library that is not strongly tied to a web framework.

json4s

2014

• https://github.com/non/jawn

• Authored by Erik Osheim (@d6)

• Backend-independent JSON parser

Jawn

• http://rapture.io/mod/json

• Authored by Jon Pretty

• Backend-independent JSON library

Rapture JSON

2015

• https://github.com/travisbrown/circe

• Authored by Travis Brown

• Port of Argonaut

Circe

2016

• https://github.com/mdedetrich/scala-json-ast

• Authored by Matthew de Detrich

• Aiming to be the common JSON AST

Scala JSON AST (SLIP-28)

• https://github.com/eed3si9n/sjson-new

• Backend-independent typeclass based JSON codec

• No macros

sjson-new

Brief history of JSON libraries

Dispatch JSON

literaljson

Lift JSON

sjsonSpray JSON

Play JSON

Argonaut

json4s

Circe

Jawn

SLIP-28 JSON

Rapture JSON

sjson-new

2009 2010 2014

Contract-first approachEugene Yokota (@eed3si9n)

2016

• “Serialization” tends to start from a programming language construct, and it generates String or byte array.

• Data binding starts with a contract or a schema of the wire format, and generates the binding in a programming language.

• XML Schema / WSDL

• Google Protocol Buffer

• Apache Thrift

• Apache Avro

• Facebook GraphQL

Serialization vs Data binding

• sealed traits

• case classes

Representing data in Scala

• sealed traits

• case classes

Representing data in Scala

Cannot evolve in a binary compatible way.

Representing data in Scalaclass Greeting(name: String) {   def copy(name: String = name): Greeting = ???   def unapply(v: Greeting): Option[String] = ??? }

class Greeting(name: String, x: Int) {   def copy(name: String = name, x: Int = x): Greeting = ???   def unapply(v: Greeting): Option[(String, Int)] = ??? }

• sealed traits

• case classes

• Cannot evolve in a binary compatible way.

• But generating equals, hash, and toString is generally useful.

Representing data in Scala

Contraband

• http://www.scala-sbt.org/contraband/

• Contraband is a description language for your datatypes and APIs, currently targeting Java and Scala.

• Based on GraphQL schema.

Contraband

Record types

package com.example @target(Scala)

## Character represents the characters in Star Wars. type Character { name: String! appearsIn: [com.example.Episode]! }

@since annotation

package com.example @target(Scala)

type Greeting { value: String! x: Int @since("0.2.0") }

Enumeration types

package com.example @target(Scala)

## Star Wars trilogy. enum Episode { NewHope Empire Jedi }

Interfaces

package com.example @target(Scala)

## Character represents the characters in Star Wars. interface Character { name: String! appearsIn: [com.example.Episode]! friends: lazy [com.example.Character] }

Record type example

package com.example @target(Scala)

type Person { name: String! age: Int }

Record type example// DO NOT EDIT MANUALLYpackage com.examplefinal class Person private (  val name: String,  val age: Option[Int]) extends Serializable {  override def equals(o: Any): Boolean = o match {    case x: Person => (this.name == x.name) && (this.age == x.age)    case _ => false  }  override def hashCode: Int = {    37 * (37 * (17 + name.##) + age.##)  }  override def toString: String = {    "Person(" + name + ", " + age + ")"

Record type example  override def toString: String = {    "Person(" + name + ", " + age + ")"  }  protected[this] def copy(name: String = name, age: Option[Int] = age): Person = {    new Person(name, age)  }  def withName(name: String): Person = {    copy(name = name)  }  def withAge(age: Option[Int]): Person = {    copy(age = age)  }  def withAge(age: Int): Person = {    copy(age = Option(age))

Record type example

object Person {  def apply(name: String, age: Option[Int]): Person = new Person(name, age)  def apply(name: String, age: Int): Person = new Person(name, Option(age))}

Record type example> val x = Person("Alice", 20)> x.withAge(21)

Contraband can derive sjson-new codecs

JSON codec generation

package com.example @target(Scala)

type Person { name: String! age: Int }

JSON codec generationpackage generatedimport _root_.sjsonnew.{ deserializationError, serializationError, Builder, JsonFormat, Unbuilder }trait PersonFormats { self: sjsonnew.BasicJsonProtocol =>  implicit lazy val personFormat: JsonFormat[_root_.Person] = new JsonFormat[_root_.Person] {    override def read[J](jsOpt: Option[J], unbuilder: Unbuilder[J]): _root_.Person = {      jsOpt match {        case Some(js) =>          unbuilder.beginObject(js)          val name = unbuilder.readField[String]("name")          val age = unbuilder.readField[Option[Int]]("age")          unbuilder.endObject()          _root_.Person(name)

JSON codec generationscala> import sjsonnew.support.scalajson.unsafe.{ Converter, CompactPrinter, Parser }scala> import com.example.codec.CustomJsonProtocol._scala> import com.example.Person

scala> val p = Person("Bob", 20)p: com.example.Person = Person(Bob, 20)

scala> val j = Converter.toJsonUnsafe(p)j: scala.json.ast.unsafe.JValue = JObject([Lscala.json.ast.unsafe.JField;@6731ad72)

scala> val s = CompactPrinter(j)s: String = {"name":"Bob","age":20}

scala> val x = Parser.parseUnsafe(s)x: scala.json.ast.unsafe.JValue =

JSON codec generationscala> val s = CompactPrinter(j)s: String = {"name":"Bob","age":20}

scala> val x = Parser.parseUnsafe(s)x: scala.json.ast.unsafe.JValue = JObject([Lscala.json.ast.unsafe.JField;@7331f7f8)

scala> val q = Converter.fromJsonUnsafe[Person](x)q: com.example.Person = Person(Bob, 20)

scala> assert(p == q)

• http://www.scala-sbt.org/contraband/

• Contraband is a description language for your datatypes and APIs, currently targeting Java and Scala.

• Low-tech metaprogramming (code generation)

• Binary compatible evolution of pseudo case class.

• Auto derivation of backend-independent JSON codec.

Contraband

• An opportunity for datatype-generic programming.

• For example, one backend is Int

Contraband

• An opportunity for datatype-generic programming.

• For example, one backend is Int (Murmurhash support)

• Builder API is used to build one-way hash.

Contraband