Writing Docker monitoring agent with Go
-
Upload
naoki-ainoya -
Category
Software
-
view
1.584 -
download
0
Transcript of Writing Docker monitoring agent with Go
Writing Docker monitoring agent with Go
!ainoya
About me
• Naoki Ainoya(!ainoya)
• iOS(new!)/Server/Infra engineer
• Swift/Scala/Go/Docker
• Recruit Marketing Partners Co, Ltd.
A tiny deployment pipeline tool Written in Go
!walter-cd/walter
Open source project
Introduction of Fune• An event emitter triggered by Docker Event API
• The agent process (fune-agent) communicates with docker daemon via its socket
• https://github.com/ainoya/fune
Docker daemonFu
ne a
gent
Cont
tain
er A
Cont
tain
er B
Cont
tain
er C
Nginx/Redis
vulcand(etcd)
Slack notification etc..
Listens docker events API
notifies each containers are created/died/stopped/started
Emits pluggable actions
Your arbitrary docker ecosystem
ECS/k8s/fleet/etc..
Docker Host
Deploy containers
Overview of Fune
• Docker Events API provides a lifecycle of container as JSON format
• create/start/die/stop
Overview of Fune
fune-agent listens these events
Docker daemonFu
ne a
gent
Cont
tain
er A
Cont
tain
er B
Cont
tain
er C
Listens docker events API
notifies each containers are created/died/stopped/started
Your arbitrary docker ecosystem
ECS/k8s/fleet/etc..
Docker Host
Deploy containers
Overview of Fune
fune-agent emits Action when received Docker events from event
Docker daemonFu
ne a
gent
Cont
tain
er A
Cont
tain
er B
Nginx/Redis
vulcand(etcd)
Slack notification etc..
Listens docker events API
notifies each containers are created/died/stopped/started
Emits pluggable actions
Your arbitrary docker ecosystem
ECS/k8s/fleet/etc..
Docker Host
Deploy containers
Overview of Fune
Docker daemonFu
ne a
gent
Cont
tain
er A
Cont
tain
er B
Nginx/Redis
vulcand(etcd)
Slack notification etc..
Listens docker events API
notifies each containers are created/died/stopped/started
Emits pluggable actions
Your arbitrary docker ecosystem
ECS/k8s/fleet/etc..
Docker Host
Deploy containers
• Action, for example;
• Set/Del a container information to Redis as FQDN/IP:Port pair for proxy
• Slack notification
• More details are next time :)
Today I talk about:
• Some tips I learned with writing fune
• Studied a lot of stuffs from coreos/etcd
Setup project
• Standard structure for working with Godeps correctly
• /src/github.com/ainoya/fune
./${GOPATH}/src!"" 9fans.net# %"" go!"" code.google.com# %"" p!"" github.com# !"" ainoya/fune# !"" barakmich# !"" coreos etc...
Godeps for the package dependencies
• Using Godeps once, all dependencies copied into directory and rewrite import statement
• Then you need Godeps command only if package dependency is added
import "github.com/fsouza/go-dockerclient"// after `Godep save -r ./…`import "github.com/ainoya/fune/Godeps/_workspace/
src/github.com/fsouza/go-dockerclient"
Godeps for the package dependencies
• All dependencies are included in git repos
• But it makes dirty commit log!!
• Keep off its dirtiness by split commits :)
How about vendor feature in go1.5?
• go1.5 vendor feature have some problems yet
• ex) go tool "./…" doesn't works well
• https://github.com/golang/go/issues/11659
• See also ) http://deeeet.com/writing/2015/06/26/golang-dependency-vendoring/
Write tests
• Rerun automatically if files are changed
• rerun is awesome! (but ruby tool gem)
rerun -x -c -d directory1, directory2,... \—pattern '**/*.{go}' -- go test ./...
Be careful about race condition
• Test with "-‐race" option if you often use goroutine in your code
• "-‐cpu N" option may reveal race condition you didn't expected
• I use "go test ./… -‐race -‐cpu 1,2,4" inside test script
WARNING: DATA RACEWrite by goroutine 8: sync.raceWrite() /usr/local/Cellar/go/1.5/libexec/src/sync/race.go:41 +0x2e sync.(*WaitGroup).Wait() /usr/local/Cellar/go/1.5/libexec/src/sync/waitgroup.go:124 +0xf9 github.com/ainoya/fune/emitter.TestBroadCast() /Users/ainoya/.go/src/github.com/ainoya/fune/gopath/src/github.com/ainoya/fune/emitter/emitter_test.go:57 +0x444 testing.tRunner() /usr/local/Cellar/go/1.5/libexec/src/testing/testing.go:456 +0xdc
Is Go1.5 compilation time slower than 1.4?
• "go test" obviously takes time much longer
• It's probably because of GoGo compiler performanceRef) Performance section of https://golang.org/doc/go1.5
• It should be improved with Go1.6
Channel tips
• coreos/etcd implements a shutdown feature with using close(channel)type Server struct { name string channel chan struct{} done chan struct{} stop chan struct{} }
func main() { done := make(chan struct{}) stop := make(chan struct{})
s := &Server{done: done, stop: stop, hoge: hoge}
go s.run()
s.channel <-‐ struct{}{}
osutil.RegisterInterrputHandler(s.Stop) <-‐s.done }
func (s *Server) run() { defer func() { close(s.done) }() for { select { case <-‐s.channel: fmt.Println("do something") case <-‐s.stop: return } }
}
func (s *S) Stop() { select { case s.stop <-‐ struct{}{}: case <-‐s.done: return } <-‐s.done }
Channel tips
• coreos/etcd implements a shutdown feature with using close(channel)type Server struct { name string channel chan struct{} done chan struct{} stop chan struct{} }
func main() { done := make(chan struct{}) stop := make(chan struct{})
s := &Server{done: done, stop: stop, hoge: hoge}
go s.run()
s.channel <-‐ struct{}{}
osutil.RegisterInterrputHandler(s.Stop) <-‐s.done }
func (s *Server) run() { defer func() { close(s.done) }() for { select { case <-‐s.channel: fmt.Println("do something") case <-‐s.stop: return } }
}
func (s *S) Stop() { select { case s.stop <-‐ struct{}{}: case <-‐s.done: return } <-‐s.done }
Channel tips
• coreos/etcd implements a shutdown feature with using close(channel)type Server struct { name string channel chan struct{} done chan struct{} stop chan struct{} }
func main() { done := make(chan struct{}) stop := make(chan struct{})
s := &Server{done: done, stop: stop, hoge: hoge}
go s.run()
s.channel <-‐ struct{}{}
osutil.RegisterInterrputHandler(s.stop) <-‐s.done }
func (s *Server) run() { defer func() { close(s.done) }() for { select { case <-‐s.channel: fmt.Println("do something") case <-‐s.stop: return } }
}
func (s *S) Stop() { select { case s.stop <-‐ struct{}{}: case <-‐s.done: return } <-‐s.done }
Channel tips
• coreos/etcd implements a shutdown feature with using close(channel)type Server struct { name string channel chan struct{} done chan struct{} stop chan struct{} }
func main() { done := make(chan struct{}) stop := make(chan struct{})
s := &Server{done: done, stop: stop, hoge: hoge}
go s.run()
s.channel <-‐ struct{}{}
osutil.RegisterInterrputHandler(s.Stop) <-‐s.done }
func (s *Server) run() { defer func() { close(s.done) }() for { select { case <-‐s.channel: fmt.Println("do something") case <-‐s.stop: return } }
}
func (s *S) Stop() { select { case s.stop <-‐ struct{}{}: case <-‐s.done: return } <-‐s.done }
Channel tips
• coreos/etcd implements a shutdown feature with using close(channel)type Server struct { name string channel chan struct{} done chan struct{} stop chan struct{} }
func main() { done := make(chan struct{}) stop := make(chan struct{})
s := &Server{done: done, stop: stop, hoge: hoge}
go s.run()
s.channel <-‐ struct{}{}
osutil.RegisterInterrputHandler(s.Stop) <-‐s.done }
func (s *Server) run() { defer func() { close(s.done) }() for { select { case <-‐s.channel: fmt.Println("do something") case <-‐s.stop: return } }
}
func (s *S) Stop() { select { case s.stop <-‐ struct{}{}: case <-‐s.done: return } <-‐s.done }
Test goroutine easily"s.done" channel makes testing goroutine easily
// https://github.com/coreos/etcd/blob/master/etcdserver/server_test.go#L1031 // TestPublishRetry tests that publish will keep retry until success. func TestPublishRetry(t *testing.T) { n := &nodeRecorder{} srv := &EtcdServer{ cfg: &ServerConfig{TickMs: 1}, r: raftNode{Node: n}, w: &waitRecorder{}, done: make(chan struct{}), reqIDGen: idutil.NewGenerator(0, time.Time{}), } time.AfterFunc(500*time.Microsecond, func() { close(srv.done) }) srv.publish(10 * time.Nanosecond)
Run go as Docker container• Go single binary with "SCRATCH"
• You can run a go stuff inside extremely simple container!
FROM scratch EXPOSE 8080 COPY your-‐golang-‐app /app/your-‐golang-‐app ENV PATH=/app:$PATH
ENTRYPOINT ["/app/your-‐golang-‐app"] CMD ["-‐some-‐option=haha"]
Round up
• Learned a lot of stuffs from well-knowned products (coreos/etcd)
• project structure, test, golang-way
See you next kyobashi.*!
• 9/7: kyobashi.dex
• 9/16: potatotips
• ???: kyobashi.???