Advanced Patterns with io.ReadWriter
-
Upload
weaveworks -
Category
Software
-
view
112 -
download
3
Transcript of Advanced Patterns with io.ReadWriter
![Page 1: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/1.jpg)
Advanced Patterns with io.ReadWriter
18 Aug 2016 Paul Bellamy
Software Engineer, Weaveworks
![Page 2: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/2.jpg)
![Page 3: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/3.jpg)
![Page 4: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/4.jpg)
Basics - What’s Available in io, bufio, & ioutil?
![Page 5: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/5.jpg)
Basics - What’s Available in io, bufio, & ioutil?
type Reader interface { Read(p []byte) (n int, err error) }
type Writer interface { Write(p []byte) (n int, err error) }
![Page 6: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/6.jpg)
Basics - What’s Available in io, bufio, & ioutil?Tools for working with ReadWriters:
• io.Copy
• io.LimitReader
• io.TeeReader
• bufio.Scanner
• ioutil.Discard
Loads more!
![Page 7: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/7.jpg)
Basics - Composition
![Page 8: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/8.jpg)
Basics - Composition
var r io.Reader r = strings.NewReader("1234567890") r = io.LimitReader(r, 5)
io.Copy(os.Stdout, r)
Setup:
Output:
![Page 9: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/9.jpg)
Basics - Composition
var r io.Reader r = strings.NewReader("1234567890") r = io.LimitReader(r, 5)
io.Copy(gzip.NewWriter(os.Stdout), r)
Setup:
Output:
![Page 10: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/10.jpg)
Example 1 - HTTP Chunking
![Page 11: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/11.jpg)
Example 1 - HTTP ChunkingLet's transparently proxy a chunked HTTP in a stream.
![Page 12: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/12.jpg)
Example 1 - HTTP ChunkingLet's transparently proxy a chunked HTTP in a stream.
Proxy
Client
![Page 13: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/13.jpg)
Example 1 - HTTP Chunking4\r\n Wiki\r\n 5\r\n pedia\r\n E\r\n in\r\n \r\n chunks.\r\n 0\r\n Date: Sun, 06 Nov 1994 08:49:37 GMT\r\n Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\r\n \r\n
![Page 14: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/14.jpg)
Example 1 - HTTP ChunkingWikipedia in
chunks.
![Page 15: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/15.jpg)
io.Copy
Solution 1
io.Copy(clientWriter, responseBody)
Obvious solution with io.Copy
responseBody clientWriter
![Page 16: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/16.jpg)
io.Copy
Solution 1
io.Copy(clientWriter, responseBody)
Obvious solution with io.Copy
responseBody clientWriter
Problems:
• Won’t let us process the Trailers separately
• We need to stop after the body
![Page 17: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/17.jpg)
io.Copy
Solution 2
io.Copy( clientWriter, httputil.NewChunkedReader(responseBody), ) parseTrailers(responseBody)
Better solution with httputil.ChunkedReader
responseBody clientWriterChunkedReader
![Page 18: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/18.jpg)
io.Copy
Solution 2
Problems:
• Strips out the chunking data
• Can’t validate the MD5
Better solution with httputil.ChunkedReader
responseBody clientWriterChunkedReader
![Page 19: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/19.jpg)
io.CopySolution 3
digest := md5.New() io.Copy(digest, httputil.NewChunkedReader( io.TeeReader( responseBody, clientWriter, ), ), ) parseTrailers(responseBody)
responseBody
clientWriter
ChunkedReader
TeeReader
digest
![Page 20: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/20.jpg)
io.CopySolution 3
io.Copy(ioutil.Discard, httputil.NewChunkedReader( io.TeeReader( responseBody, clientWriter, ), ), ) parseTrailers(responseBody)
responseBody
clientWriter
ChunkedReader
TeeReader
ioutil.Discard
![Page 21: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/21.jpg)
Example 2 - Multiplexing
![Page 22: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/22.jpg)
Example 2 - MultiplexingRun subcommands, multiplex the output, give each a unique prefix.
echo “Hello\nWorld!” make all git status
[echo] Hello, [make] make: *** No rule to make target. [echo] World! [git] On branch master [git] Your branch is up-to-date [git] nothing to commit
![Page 23: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/23.jpg)
Example 2 - Multiplexing
![Page 24: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/24.jpg)
Example 2 - MultiplexingStarting up the commandscommands := [][]string{ {"echo", "Hello,\nWorld!"}, {"make", "all"}, {"git", "status"}, }
for _, cmd := range commands { c := exec.Command(cmd[0], cmd[1])
// Wire up our prefix writer prefixedOutput := prefixingWriter(cmd[0], os.Stdout) c.Stdout = prefixedOutput c.Stderr = prefixedOutput
// Start each in a goroutine go func() { done <- c.Run() }() }
![Page 25: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/25.jpg)
Example 2 - MultiplexingprefixingWriterfunc prefixingWriter(prefix string, output io.Writer) io.Writer
![Page 26: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/26.jpg)
Example 2 - MultiplexingSplitting streams into sections with bufio.Scanner input := ...
// Iterate over each line scanner := bufio.NewScanner(input) scanner.SplitFunc(bufio.ScanLines)
for scanner.Scan() {
// Write the prefix into the output fmt.Fprintf(output, "[%s] ", prefix)
// Copy the line output.Write(scanner.Bytes())
// Re-add a newline (scanner removes it) fmt.Fprint(output, "\n")
}
![Page 27: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/27.jpg)
Example 2 - MultiplexingSplitting streams into sections with bufio.Scanner input := ...
// Iterate over each line scanner := bufio.NewScanner(input) scanner.SplitFunc(bufio.ScanLines)
for scanner.Scan() {
// Write the prefix into the output fmt.Fprintf(output, "[%s] ", prefix)
// Copy the line output.Write(scanner.Bytes())
// Re-add a newline (scanner removes it) fmt.Fprint(output, "\n")
}
![Page 28: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/28.jpg)
Example 2 - MultiplexingSplitting streams into sections with bufio.Scanner input := ...
// Iterate over each line scanner := bufio.NewScanner(input) scanner.SplitFunc(bufio.ScanLines)
for scanner.Scan() {
// Write the prefix into the output fmt.Fprintf(output, "[%s] ", prefix)
// Copy the line output.Write(scanner.Bytes())
// Re-add a newline (scanner removes it) fmt.Fprint(output, "\n")
}
![Page 29: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/29.jpg)
Example 2 - MultiplexingSplitting streams into sections with bufio.Scanner input := ...
// Iterate over each line scanner := bufio.NewScanner(input) scanner.SplitFunc(bufio.ScanLines)
for scanner.Scan() {
// Write the prefix into the output fmt.Fprintf(output, "[%s] ", prefix)
// Copy the line output.Write(scanner.Bytes())
// Re-add a newline (scanner removes it) fmt.Fprint(output, "\n")
}
![Page 30: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/30.jpg)
Example 2 - MultiplexingSplitting streams into sections with bufio.Scanner input := ...
// Iterate over each line scanner := bufio.NewScanner(input) scanner.SplitFunc(bufio.ScanLines)
for scanner.Scan() {
// Write the prefix into the output fmt.Fprintf(output, "[%s] ", prefix)
// Copy the line output.Write(scanner.Bytes())
// Re-add a newline (scanner removes it) fmt.Fprint(output, "\n")
}
![Page 31: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/31.jpg)
Example 2 - MultiplexingSplitting streams into sections with bufio.Scanner input := ...
// Iterate over each line scanner := bufio.NewScanner(input) scanner.SplitFunc(bufio.ScanLines)
for scanner.Scan() {
// Write the prefix into the output fmt.Fprintf(output, "[%s] ", prefix)
// Copy the line output.Write(scanner.Bytes())
// Re-add a newline (scanner removes it) fmt.Fprint(output, "\n")
}
![Page 32: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/32.jpg)
Example 2 - MultiplexingSplitting streams into sections with bufio.Scanner input := ...
// Iterate over each line scanner := bufio.NewScanner(input) scanner.SplitFunc(bufio.ScanLines)
for scanner.Scan() {
// Write the prefix into the output fmt.Fprintf(output, "[%s] ", prefix)
// Copy the line output.Write(scanner.Bytes())
// Re-add a newline (scanner removes it) fmt.Fprint(output, "\n")
}
![Page 33: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/33.jpg)
Example 2 - MultiplexingWhat is input?
• bufio.Scanner expects a io.Reader
• exec.Cmd expects a io.Writer
![Page 34: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/34.jpg)
goroutine
Example 2 - Multiplexing
pipeWriter bufio.Scannerio.Pipe output
io.Pipe to the rescue
![Page 35: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/35.jpg)
Example 2 - Multiplexingfunc prefixingWriter(prefix string, output io.Writer) io.Writer { pipeReader, pipeWriter := io.Pipe()
scanner := bufio.NewScanner(pipeReader) scanner.SplitFunc(bufio.ScanLines)
go func() { for scanner.Scan() { fmt.Fprintf(output, "[%s] ", prefix) output.Write(scanner.Bytes()) fmt.Fprint(output, "\n") } }()
return pipeWriter }
![Page 36: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/36.jpg)
Example 2 - Multiplexingfunc prefixingWriter(prefix string, output io.Writer) io.Writer { pipeReader, pipeWriter := io.Pipe()
scanner := bufio.NewScanner(pipeReader) scanner.SplitFunc(bufio.ScanLines)
go func() { for scanner.Scan() { fmt.Fprintf(output, "[%s] ", prefix) output.Write(scanner.Bytes()) fmt.Fprint(output, "\n") } }()
return pipeWriter }
![Page 37: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/37.jpg)
Example 2 - Multiplexingfunc prefixingWriter(prefix string, output io.Writer) io.Writer { pipeReader, pipeWriter := io.Pipe()
scanner := bufio.NewScanner(pipeReader) scanner.SplitFunc(bufio.ScanLines)
go func() { for scanner.Scan() { fmt.Fprintf(output, "[%s] ", prefix) output.Write(scanner.Bytes()) fmt.Fprint(output, "\n") } }()
return pipeWriter }
![Page 38: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/38.jpg)
Example 2 - Multiplexingfunc prefixingWriter(prefix string, output io.Writer) io.Writer { pipeReader, pipeWriter := io.Pipe()
scanner := bufio.NewScanner(pipeReader) scanner.SplitFunc(bufio.ScanLines)
go func() { for scanner.Scan() { fmt.Fprintf(output, "[%s] ", prefix) output.Write(scanner.Bytes()) fmt.Fprint(output, "\n") } }()
return pipeWriter }
![Page 39: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/39.jpg)
Example 2 - Multiplexingfunc prefixingWriter(prefix string, output io.Writer) io.Writer { pipeReader, pipeWriter := io.Pipe()
scanner := bufio.NewScanner(pipeReader) scanner.SplitFunc(bufio.ScanLines)
go func() { for scanner.Scan() { fmt.Fprintf(output, "[%s] ", prefix) output.Write(scanner.Bytes()) fmt.Fprint(output, "\n") } }()
return pipeWriter }
![Page 40: Advanced Patterns with io.ReadWriter](https://reader034.fdocuments.us/reader034/viewer/2022042502/5876fcd51a28abf3398b689f/html5/thumbnails/40.jpg)
Thank you18 Aug 2016
Paul Bellamy
Software Engineer, Weaveworks
github.com/paulbellamy/golanguk2016
@pyrhho