Scala in Nokia Places API

Lukasz Balamut - Places API Team

17 April 2013

Purpose of this talk

I Why are we spending time migrating to a new language?I How are we doing that?I What have we learned?

and of course: we are hiring.

Our scala story.

Our migration in lines of code

We’ve discovered more and more things on the way.

The Concordion tests problemIn June 2011 we had problems with Concordion (HTML based acceptance testframework),

I hard to integrateI HTML tests were not readable

Solution:

I Multi-line Strings in ScalaI Continue using JUnit

val expectedBody = """

{"id":"250u09wh-83d4c5479b7c4db9836602936dbc8cb8",

"title" : "Marfil (Le)",

"alternativeNames" : [

{"name" : "Le Marfil",

"language" : "fr"

}]

}"""

The test names problem

I JUnit + camel case - not that readable for long names

@Test

def preservesSearchResponseItemOrder() { /* ... */ }

I JUnit + underscores - violates naming convention, but better to read

@Test

def preserves_search_response_item_order() { /* ... */ }

I ScalaTest

test("preserves search response’s item order") { /* ... */ }

It is supported by tools we use and produces nice output when run (as a bonus)

[info] DiscoverAroundAcceptanceTest:

[info] - calls nsp with 15Km radius

[info] - preserves search response’s item order

[info] - when requesting zero results, no request to search is made

Lift-Json

I JSON (de)serialisation can be done in really nice way

e.g. getting place id:

{"place": {

"a_id": "276u33db-6f084959f97c45bc9db26348bafd4563"

}}

Also there is a generic way of parsing json, that gives you XPath like access:

val id = (json \ "place" \ "a_id").extract[String]

I Lift-json provides also easy and efficient case class JSON (de)serialisation,hance case classes

Scala case classes in the modelWe model our upstream and downstream datamodels as case classes.

case class Place (

name: String,

placeId: PlaceID

//(...)

)

I Each case class is a proper immutable data type - the new Java Bean ;),I A lot of boilerplate code is generated by the compiler e.g.:

I accessors,I equals/hashCode,I copy methods,I toStringI apply/unapply (for pattern maching),

After decompiling the scala code above we will end up with 191 lines of Java:

package pbapi.model.api;

import scala.*;

import scala.collection.Iterator;

import scala.runtime.BoxesRunTime;

import scala.runtime.ScalaRunTime$;

public class Place

implements Product, Serializable

{

public static final Function1 tupled() {return Place$.MODULE$.tupled();

}

public static final Function1 curry() {return Place$.MODULE$.curry();

}

public static final Function1 curried() {return Place$.MODULE$.curried();

}

public Iterator productIterator() {return scala.Product.class.productIterator(this);

}

public Iterator productElements() {return scala.Product.class.productElements(this);

}

public String name() {return name;

}

public PlaceID placeId() {return placeId;

}

public Place copy(String name, PlaceID placeId) {return new Place(name, placeId);

}

public PlaceID copy$default$2() {return placeId();

}

public String copy$default$1() {return name();

}

public int hashCode() {return ScalaRunTime$.MODULE$._hashCode(this);

}

public String toString() {return ScalaRunTime$.MODULE$._toString(this);

}

public boolean equals(Object obj) {if(this == obj) goto _L2; else goto _L1

_L1:

Object obj1 = obj;

if(!(obj1 instanceof Place)) goto _L4; else goto _L3

_L3:

String name$6;

PlaceID placeId$4;

Place place1 = (Place)obj1;

String s = place1.name();

PlaceID placeid = place1.placeId();

name$6 = s;

placeId$4 = placeid;

if(gd22$1(name$6, placeId$4) ? ((Place)obj).canEqual(this) : false) goto _L2; else goto _L5

_L4:

if(false) goto _L2; else goto _L5

_L2:

true;

goto _L6

_L5:

false;

_L6:

return;

}

public String productPrefix() {return "Place";

}

public int productArity() {return 2;

}

public Object productElement(int i) {int j = i;

j;

JVM INSTR tableswitch 0 1: default 24

// 0 39

// 1 46;

goto _L1 _L2 _L3

_L1:

throw new IndexOutOfBoundsException(BoxesRunTime.boxToInteger(i).toString());

_L2:

name();

goto _L4

_L3:

placeId();

_L4:

return;

}

public boolean canEqual(Object obj) {return obj instanceof Place;

}

private final boolean gd22$1(String s, PlaceID placeid) {s;

String s1 = name();

if(s != null) goto _L2; else goto _L1

_L1:

JVM INSTR pop ;

if(s1 == null) goto _L4; else goto _L3

_L2:

s1;

equals();

JVM INSTR ifeq 57;

goto _L4 _L3

_L4:

placeid;

PlaceID placeid1 = placeId();

if(placeid != null) goto _L6; else goto _L5

_L5:

JVM INSTR pop ;

if(placeid1 == null) goto _L7; else goto _L3

_L6:

placeid1;

equals();

JVM INSTR ifeq 57;

goto _L7 _L3

_L7:

true;

goto _L8

_L3:

false;

_L8:

return;

}

public Place(String name, PlaceID placeId) {this.name = name;

this.placeId = placeId;

super();

scala.Product.class.$init$(this);

}

private final String name;

private final PlaceID placeId;

}

package pbapi.model.api;

import scala.*;

import scala.runtime.AbstractFunction2;

public final class Place$ extends AbstractFunction2

implements ScalaObject, Serializable

{

public final String toString() {return "Place";

}

public Option unapply(Place x$0) {return ((Option) (x$0 != null ? new Some(new Tuple2(x$0.name(), x$0.placeId())) : None$.MODULE$));

}

public Place apply(String name, PlaceID placeId) {return new Place(name, placeId);

}

public Object readResolve() {return MODULE$;

}

private Place$() {}

public static final Place$ MODULE$ = this;

static {new Place$();

}}

Scala case classes - Configuration

I Easy to update configurationI Easy to compare environments’ configuration

case class HttpClientConfig(

serviceName: String,

serviceBaseUrl: URL,

readTimeout: Int = 0,

connectionTimeout: Int,

proxy: Option[ProxyConfig] = None,

oauthKeys: Option[OAuthConfig] = None,

displayUrl: Option[String] = None

)

"httpConfig":{"serviceName":"recommendations",

"serviceBaseUrl":"http://nose.svc.ovi.com/rest/v1/recommendations/nearby/",

"readTimeout":4000,

"connectionTimeout":500,

"displayUrl":"http://nose.svc.ovi.com/rest/v1/recommendations/nearby/"

}

The Scala Type System

I A great way to express assumptionsI Have them checked by the compiler through the whole codebaseI At every point in the code know what to expect or you will be reminded by

compiler.

The Scala Type System - Options 1

I Avoid null references (see The Billion Dollar Mistake )

This will not compile!

case class UpstreamResponse ( unreliableField: Option[String] )

case class MyResponse ( reliableField: String )

def transform(upstream: UpstreamResponse) =

MyResponse(

reliableField = upstream.unreliableField //<-- incompatible type

)

The Scala Type System - Options 2

But this will:

case class UpstreamResponse ( unreliableField: Option[String] )

case class MyResponse ( reliableField: String )

def transform(upstream: UpstreamResponse) =

MyResponse(

reliableField = upstream.unreliableField.getOrElse("n/a") // enforced by compiler

)

The Scala Type System - Options 3

how we were learning e.g.: to transform string when it is defined

def transform(str: String): String

reliableField =

if (upstream.unreliableField.isDefined)

transform(upstream.unreliableField.get)

else "n/a"

reliableField =

upstream.unreliableField match {case Some(f) => transform(f)

case None => "n/a"

}

reliableField =

upstream.unreliableField.map(transform) | "n/a"

The Scala Type System - Standard Types

I Immutable Map, List, Sets

val map: Map[String, String] = Map("a" -> "a", "b" -> "b")

val list: List[String] = "a" :: "b" :: Nil

I Tuples to express Pairs, Triplets etc.

val pair: Pair[String, Int] = ("a", 1)

val (a, b) = pair // defines a:String and b:Int

The Scala Type System - error handlingI Types can help with exceptional cases

val parsed: Validation[NumberFormatException, Int] = someString.parseInt

val optional: Option[Int] = parsed.toOption

val withDefault: Int = optional.getOrElse(0) //if there was parsing problem 0

I Exception to Option

val sub: Option[String] =

allCatch.opt(line.substring(line.indexOf("{\"")))

I or Either

val parsed: Either[Throwable, Incident] =

allCatch.either(new Incident(parse(jsonString))

and handle it later

parsed match {case Right(j) => Some(j)

case Left(t) => {println("Parse error %s".format(t.getMessage))

None

}}

The Scala Type System - functions

I we defer translation (Translated) and text rendering (FormattedText) toserialisation time using function composition so we don’t need to pass usercontext through the whole stack e.g.:

case class Place (

//(...)

attribution: Option[Translated[FormattedText]],

//(...)

)

case class Translated[A](translator: TransEnv => A) {def apply(env: TransEnv): A = translator(env)

def map[B](f: A => B): Translated[B] =

new Translated[B](env => f(apply(env)))

//(...)

}

XML literals

XML can be part of the code and compiler is checking syntax of it:

def toKmlCoordinates(style: String): String =

(<Placemark>

<name>{"bbox " + this}</name><styleUrl>{style}</styleUrl><Polygon>

<outerBoundaryIs>

<LinearRing>

<coordinates>

{west + "," + north + ",0"}{east + "," + north + ",0"}{east + "," + south + ",0"}{west + "," + south + ",0"}{west + "," + north + ",0"}

</coordinates>

</LinearRing>

</outerBoundaryIs>

</Polygon>

</Placemark>).toString

ScalacheckAutomatic testing of assumptions, using set of generated values.

e.g. the code below tests if

BBox.parse(a.toString) == a

is true for any a

val generator: Gen[BBox] =

for {s <- choose(-90.0, 90.0)

n <- choose(-90.0, 90.0)

if (n > s)

w <- choose(-180.0, 180.0)

e <- choose(-180.0, 180.0)

} yield new BBox(w, s, e, n)

property("parse . toString is identity") {check {

(a: BBox) =>

BBox.parse(a.toString) == a

}}

may yield this failing test message after several evaluations.

GeneratorDrivenPropertyCheckFailedException was thrown during property evaluation.

(BBoxTest.scala:32)

Falsified after 2 successful property evaluations.

Location: (BBoxTest.scala:32)

Occurred when passed generated values (

arg0 = 33.50207944036654,-63.96980425691198,59.04945970748918,8.222590173180834

)

Scala (Scalding) in our analytics jobse.g. popular places job

class PopularPlacesJob(args: Args) extends AccessLogJob(args) {

implicit val mapMonoid = new MapMonoid[String, Long]()

val ord = implicitly[Ordering[(Long, String)]].reverse

readParseFilter()

.flatMap(’entry -> ’ppid) {entry:AccessLogEntry => entry.request.ppid

}.mapTo((’entry, ’ppid) -> (’ppid, ’time, ’app)) {

arg: (AccessLogEntry, String) => (arg._2, arg._1.dateTime, arg._1.appId.getOrElse("?"))

}.groupBy((’ppid, ’app)) {

_.min(’time -> ’minTime)

.max(’time -> ’maxTime)

.size(’num)

}.groupBy((’ppid)) {

_.min(’minTime)

.max(’maxTime)

.sum(’num)

.mapPlusMap(((’app, ’num) -> ’apps))

{ Map(_:(String, Long)) }{ _.toList

.map(a => (a._2, KnownClients.appName(a._1, "N/A")))

.sorted(ord)

.mkString(";")

}}.reverseSortAndLimit(’num)

.writeWithHeader(args("output"))

}

Not so good in Scala

I Almost whole team needed to learn the new language - luckily we haveToralf ;)

I Tools are not as mature as those in Java (but they are getting better veryfast)

I Longer compilation, compiler has a lot more to do (e.g. type inference)

I Multiple inheritance (traits)

I Operators

I changes in every language release

What we have learned, discovered?

I When done right - it’s possible to painlessly migrate code to newlanguage, developing new feautres in the same time

I How to use good typesystem and enjoy it

I Take advantage form whole great immutable world of painlessprogramming

I Use functions as first class citizens of language

Plans

I Scala 2.10I Run Transformations in Futures, Validations etc.I Akka actors, Futures compositionI Kill last Java files?I Stop using Spring

Thank you!

I project site:places.nlp.nokia.com

I My contact:lukasz.balamut@nokia.com@lbalamut

I open position in our team: