The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31,...
Transcript of The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31,...
![Page 1: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/1.jpg)
Copyright © 2019 HashiCorp
Adventures in Writing Distributed Systems in Go:
The Good, the Bad, and the Ugly Code in HashiCorp Nomad
![Page 2: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/2.jpg)
Michael Schurter@schmichael
Nomad Team Lead at HashiCorp
![Page 4: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/4.jpg)
HashiCorp Products
![Page 5: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/5.jpg)
Multi-Cloud DevOps Infrastructure Stuff
Provision
Run
Connect
Secure
![Page 6: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/6.jpg)
What is
![Page 8: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/8.jpg)
![Page 9: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/9.jpg)
4+ yearsStarted May 31, 2015
v0.1 September 28, 2015
2.5k+ testsUnit, integration, e2e, xplat
Excluding table/sub tests
244k+ LoC1.1M+ counting vendoring
Go code
510 depsStill vendoring
Modules soon I promise.
Nomadby the Numbers
![Page 10: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/10.jpg)
Legend
Good Gopher Bad Gopher Ugly Gopher
![Page 11: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/11.jpg)
Single Binary; Multiple Platforms
https://releases.hashicorp.com/nomad/0.9.5/
![Page 12: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/12.jpg)
Not all platformsAlpineARMCGO
![Page 13: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/13.jpg)
go get github.com/hashicorp/nomad
This implies go build, install, etc work as well.
![Page 14: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/14.jpg)
go test ./...
E2E tests enabled via env var; all others run by default.
![Page 15: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/15.jpg)
go test ./...Slow.
Eats all your CPU.Many tests require root which means we may destroy your computer.
![Page 16: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/16.jpg)
Use Vagrant for Client Tests
![Page 17: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/17.jpg)
TERMINAL
$ vagrant up
Bringing machine 'linux' up with 'virtualbox' provider...
…
$ vagrant ssh
Welcome to Ubuntu 16.04.6 LTS (GNU/Linux 4.4.0-131-generic x86_64)
....
Last login: Fri Jul 19 17:55:15 2019 from 10.0.2.2
vagrant@linux:/opt/gopath/src/github.com/hashicorp/nomad$
VagrantRoot Vagrant box suitable for dangerous tests.
![Page 18: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/18.jpg)
TERMINAL
$ export AWS...
$ terraform apply
…
$ NOMAD_E2E=1 go test -v
=== RUN TestE2E
=== RUN TestE2E/Affinity
=== RUN TestE2E/Affinity/*affinities.BasicAffinityTest
...
TerraformTerraform for E2E clusters.
Many work locally too!
![Page 19: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/19.jpg)
Quick and easy network proxying
Takes some bits from over here and put them over there.
![Page 20: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/20.jpg)
Network Namespaces in 0.10
![Page 21: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/21.jpg)
Proxy out of Network Namespace
![Page 22: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/22.jpg)
Quick and easy network proxying~100 lines of code
https://github.com/hashicorp/nomad/blob/v0.10.0-beta1/client/allocrunner/consulsock_hook.go#L202-L319
![Page 23: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/23.jpg)
Copy paste copy paste copy paste
Normally I don’t miss metaprogramming but ACLs...
![Page 24: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/24.jpg)
![Page 25: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/25.jpg)
![Page 26: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/26.jpg)
![Page 27: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/27.jpg)
![Page 28: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/28.jpg)
With great concurrency…Comes great complexity.
https://github.com/hashicorp/nomad/pull/6082
![Page 29: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/29.jpg)
Contexts are great...
![Page 30: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/30.jpg)
...until you need coordinated shutdowns.▪ Don’t leak goroutines!▪ Causes memory leaks.▪ Causes race detector failures in tests:
panic: Log in goroutine after Test... has completed
![Page 32: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/32.jpg)
There are worse fates than locks...
![Page 33: The Good, the Bad, and the Ugly Code in HashiCorp Nomad · 2019-10-03 · 4+ years Started May 31, 2015 v0.1 September 28, 2015 2.5k+ tests Unit, integration, e2e, xplat Excluding](https://reader033.fdocuments.us/reader033/viewer/2022050108/5f45db276916a8634b2be452/html5/thumbnails/33.jpg)
schmichael2017
Locks are evil!Communicate by passing messages!