Logging in Scala

John Nestor 47 Degrees

www.47deg.com

August 9, 2016

147deg.com

47deg.com © Copyright 2016 47 Degrees

Outline

• Introduction

• Architecture

• Features

• Implementation

• Demos

• Cross Service Log Aggregation

2

Introduction

3

47deg.com © Copyright 2016 47 Degrees

Existing Scala Loggers

• Java loggers (used in Scala code)

• Log4j (used in Spark)

• SLF4J and Logback

• Scala loggers

• Abandoned: Configgy, Logula (Coda Hale)

• Log4s (wraps SLF4J)

• scala-logging (Heiko Seeberger) (wraps Log4j)

• Special Scala loggers

• Akka actor logging (can route to Logback)

• Play logging (uses Logback)

4

47deg.com © Copyright 2016 47 Degrees

Why a New Logger?

• Pure Scala. (No Java code)

• Richer messages (Json rather than text strings)

• Aggregate logs across multiple (micro-)services

• Clean design with rich features

5

47deg.com © Copyright 2016 47 Degrees

New Scala Logger History

• Whitepages Logger - part of a new Scala web service framework

• Persist Logger (work supported by 47 Degrees)

• clean up and improve API

• removed Whitepages specific stuff

• added complete documentation

• Full source and documentation on Github

• Maven central "com.persist" % "persist-logging_2.11" % "1.2.4"

• Apache 2 license

6

47deg.com © Copyright 2016 47 Degrees

Rich Logging Example

log.error(s”Too many values Size:${vals.size} Max:${vals.max}”)

log.error(map(“msg”->”Too many values”, ”Size”->vals.size, ”Max”->vals.max), id=currentId)

{“msg”:”Too Many Values”, “Size”: 10, “Max: “33}

7

Architecture

8

47deg.com © Copyright 2016 47 Degrees

Persist Logger Architecture

9

logging.conf

logback.xml

Scala Logger API Slf4j API

Akka Actor Logging API

Custom Logback

Appender

LogBack Custom Akka Logging Actor

Logging Actor

Stdout Appender

File Appender

Kafka Appender

Other Appenders

Logger State

Scala Logger

Control API

Features

10

47deg.com © Copyright 2016 47 Degrees

API Log Call

• Level

• Rich Message

• Id (optional)

• Per request level control

• Per request log aggregation

• Exception (optional)

• use RichException for a RichMessage

11

47deg.com © Copyright 2016 47 Degrees

Rich Messages

• String

• Int, Long, Float, Double, BigDecimal

• null, Boolean

• Seq[RichMessage]

• Map[String,RichMessage]

12

47deg.com © Copyright 2016 47 Degrees

Why Json?

• Easy to support rich metadata in addition to the message

• Easier to parse in downstream tools

• No need to manually escape strings

• Naturally supports multiline messages

• Standard text format

• But many internal possibilities

• This logger uses Persist Json (only internally)

• Rich messages use standard Scala types for in internal API

13

47deg.com © Copyright 2016 47 Degrees

Kind of Logs

• Error log

• Alternative logs

• GC

• Timing (for fine grain timing)

• Server (request-response, duration)

• Client (request, response, duration)

• User defined

14

47deg.com © Copyright 2016 47 Degrees

Standard Fields

• @category (error, gc, …)

• @host

• @service

• @severity (info, debug, warn, error, …)

• @timestamp (msec)

• class, file, line, actor

• msg

• trace (for exceptions, Json)

15

47deg.com © Copyright 2016 47 Degrees

Appenders

• Stdout

• no server/service specific fields

• color (optional)

• summary

• File (daily rotation)

• Kafka

• Custom, user defined

• Control destination

• Control contents and format

16

47deg.com © Copyright 2016 47 Degrees

Level Control

• logging.conf (default options), can override

• uses Typesafe config

• levels: API, SLF4J, Akka

• Change level via API

• Per request custom level

• Custom filters

• Based on level, and content

17

47deg.com © Copyright 2016 47 Degrees

Timing

• Fine grain timing

• Uses request ids to aggregate

• Time in microseconds

• Time log

• Can be used with concurrency (futures and actors)

18

47deg.com © Copyright 2016 47 Degrees

Other Loggers

• Captures messages from other legacy loggers that might be used in new project libraries

• SLF4J (captured via Logback appender)

• Akka Actor Logging (captured by custom Akka log handing actor)

19

47deg.com © Copyright 2016 47 Degrees

Look At Documentation

• logging.conf

• Overview

• API

20

Implementation

21

47deg.com © Copyright 2016 47 Degrees

Persist Logger Architecture

22

logging.conf

logback.xml

Scala Logger API Slf4j API

Akka Actor Logging API

Custom Logback

Appender

LogBack Custom Akka Logging Actor

Logging Actor

Stdout Appender

File Appender

Kafka Appender

Other Appenders

Logger State

Scala Logger

Control API

47deg.com © Copyright 2016 47 Degrees

Dealing with Concurrency

• The logging actor provides fully generalized sync

• Serializes output

• Handles filters

• But asking the actor on every message can be slow

• @volatile booleans for each log level associated with logger

• Note the logger itself is global state

• One copy for entire app, no need to pass a parameter everywhere: ClassLogging and ActorLogging traits.

• Must start before any messages and stop after any messages (but there is a development workaround)

• Contains log level booleans23

47deg.com © Copyright 2016 47 Degrees

Source Location

• Could find using exception stack trace at run-time

• Better solution is to use Scala macro and reflection at compile-time

24

Demos

25

47deg.com © Copyright 2016 47 Degrees

Demos

• Simple

• Actor

• Exceptions

• Request Id

• Alternative

• Other (Actor, SLF4J)

• Timing

• Filter

• Appender

26

Cross Service Log Aggregation

27

47deg.com © Copyright 2016 47 Degrees

The Problem

• Applications are being split into ever more micro-services

• Much harder to answer questions?

• What is the overall control flow for various requests?

• What was the root cause of an error?

• Where are the performance bottlenecks?

• In this talk we look at a prototype system that can be used to answer these kinds of questions

• Work in progress

28

47deg.com © Copyright 2016 47 Degrees

Related Work

• Google Dapper

• Zipkin

• Akka Tracing (wraps Zipkin)

29

47deg.com © Copyright 2016 47 Degrees

Approach

• Use Scala as much as possible

• Aggregate logs for each independent request across a set of Scala services

• Provide a basis for both

• Near real-time

• Alerts

• View aggregated logs for any request

• More comprehensive batch analysis

• Failure statistics

• Overall patterns

• Trends over time

30

47deg.com © Copyright 2016 47 Degrees

Overall Architecture

31

KafkaSpark

Streaming(Aggregation)

NoSql DB

REST Log API *

(Merge, Filter)

Services *

* Uses: Persist Service Framework

Persist Service Framework Uses:Akka HTTPPersist Json

Persist Logger

Spark Batch Analytics

47deg.com © Copyright 2016 47 Degrees

Scala REST Service

32

Server Log

Service(can be lots of these)

Error Log

Client Log

... To Other Services

Kafka(only one cluster/topic for all

logs and services)

... From Other Services

47deg.com © Copyright 2016 47 Degrees

Services

33

Serial

A

Driver

E

D

C

B

Parallel

47deg.com © Copyright 2016 47 Degrees

Aggregation (via Spark Streaming)

• Pass from client to server

• Client name

• Tracking Id

• same for a single request across all micro services

• Span Id

• unique for each call for a given Client-Server pair

• Match Tracking Id and Span Id across

• Client log

• Server log

34

47deg.com © Copyright 2016 47 Degrees

Cross Service Matching

35

Id1Span1

X

AId1

Span1X

Id1Span2

A

Id1Span3

A

AId1

Span1Span2

B

BId1

Span2A

BId1

Span3A

AId1

Span1Span3

B

Match

Match

Match

Match

Service A Service B

47deg.com © Copyright 2016 47 Degrees

Look at Sample Output

• Array of log messages

• Tree of log messages

• Filtered tree with timings

36

Questions

37