Engineering Efficiency powered by Go

Huy Do Software Engineering

LINE corp

reflect.TypeOf(Who)me := &Who { Name: "Huy Do", Location: "Tokyo", Org: "LINE corp", Misc: "admin member of Ruby Vietnam group, and co-founded kipalog.com",}

Engineering Efficiency

How to measure effectiveness?

effectiveness =

Time Saved

“We massively underinvest in this kind of work”

Peter Seibel, Twitter

How many dedicated engineers?

http://www.gigamonkeys.com/flowers/

Engineering Efficiency at LINE

We do anything, except raising your

salary!

• Building in-house deploy system • Building scale in-house monitoring system • Building / maintaining OSS CI/Repository

system

Building many tools help our colleague engineers focus in write code efficiently

https://linecorp.com/ja/career/position/786

Introduce Lineflow

LINE Development process

https://engineering.linecorp.com/ja/blog/detail/38

1. Make JIRA ticket 2. Create local branch follow TICKET-ID-feature 3. Make a Pull Request to upstream/develop 4. Release to Beta server 5. Request QA team to confirm feature 6. Cherry-pick changes to local TICKET-ID-release branch 7. Make a Pull Request to upstream/release 8. Increase artifact version 9. Create RC branch and release to staging server and confirm

integration 10. Release to Canary server and confirm no big problem 11. Release to all server 12. Close JIRA ticket

We need to follow many conventions as well

12 steps needed to release your code production!!!!

it's just too much!!!!!

But Process is needed

• Procedures protect us from stepping on another's toes

• Conventions help us inspect problem

Problem 1: Manual copy paste is just painful

design CLI tool for process automation

Lineflow goals

• Extensible • Safe • Reflect the domain • Zero dependency

Some concepts • flow

• Set of commands belong to same domain

• subcommand• Represent single execute step

workflow

subcommand

Implementation

type Command struct { Execute func(cmd *Command, args *Args) Flag flag.FlagSet

... Name string parent *Command childs map[string]*Command }

type Args struct { Executable string Command string ProgramPath string Params []string

... }

type Executor struct { commands map[string]*Command execute func(cmds []*cmd.Cmd) error }

We wrote simple wrapper for os/exec

func (r *Executor) Execute() ExecError { .... cmd := Find(args.Command) if cmd != nil { return r.Call(cmd, args) } printUsage() return newExecError(errors.New("command not exist")) }

func (r *Executor) Find(name string) *Command { return r.commands[name] }

Problem 2: I just forgot what I should do next

Command suggestion

Implementation

Define flow order

flowSpec.Next(cmdCommit). Next(cmdDevelopPullRequest). Next(cmdReleaseCherryPick). Next(cmdReleasePullRequest). Next(cmdVersionCommit). Next(cmdRcBranchCreate). Next(cmdReleaseShareTemplate). Next(cmdNewDevelopBranch)

DSL to define depedency graph

type FlowSpec struct { root *CmdNode last *CmdNode }

func (f *FlowSpec) NextOf(cmd *c.Command) (*c.Command, error)

func (f *FlowSpec) Next(cmd *c.Command) *FlowSpec

struct called FlowSpec to implement the DSL

type CmdNode struct { value *c.Command next *CmdNode prev *CmdNode } func (n *CmdNode) Prev(cn *CmdNode) { n.prev = cn cn.next = n }

func (n *CmdNode) Next(cn *CmdNode) { n.next = cn cn.prev = n }

func (n *CmdNode) GetNext() *CmdNode { return n.next }

func (n *CmdNode) GetPrev() *CmdNode { return n.prev }

Simple double linked list

What is the last thing that you've done ?

• Branch name as unique identity

• Just store every executed command in ~/.lineflow_history (same idea with .bash_history)

type History struct { fileLocation string commands []*c.Command }

func (h *History) Register(cs ...*c.Command)

func (h *History) Save(cmd *c.Command, meta string) error

func (h *History) traverseFromTop( f func(*c.Command, string) bool) error

func (h *History) Last(parent *c.Command, meta string) *c.Command

func (h *History) toHistoryString(cmd *c.Command) string

func (h *History) historyStringToCommand(s string) (*c.Command, string, error)

So basically that's how lineflow is built

Thanks to https://github.com/studio3104

lineflow + peco

Lineflow impact• Saved time

• 500 engineer * 5 minutes ~ 41 hours (per day)

• Make foundation for further internal tool intergration

• Help new comer get used to the process

What we learned• Golang is bad at writing DSL • Hard to develop plugin-style due static

code loader (compare to ruby)

How golang survive in Java shop

• IntelliJ Golang plugin is great (we can work with both java/golang in same editor)

• go tools ecosystem is great (debugger, benchmark, test tool..)

• Distribute golang code is great (we love binary)

How to build good internal product

• MUST invest resource (there are no good short-term solution)

• API, and good API • You need to be an evangelist for your

own product, or no body will use yours :(

Thank you