Post on 01-Dec-2014
description
Copyright © 2014 NTT DATA Corporation
10/15/2014Masatake Iwasaki
HTrace: Tracing in HBase and HDFS
Presentation slide for HBase Meetup on the Night before Strata/HW @ Google in Chelsea
2 Copyright © 2014 NTT DATA Corporation
What is HTrace?
https://github.com/cloudera/htrace
Tracing tool for parallel distributed systemlike Google's Dapper �
Effective for finding bottleneckEffective for code analysisLow Overhead
3 Copyright © 2014 NTT DATA Corporation
Tracing spans
Span represents traced processing unit and timeSpans have parent-‐‑‒child relationshipPassing tracing info along with RPC
Time
Span A trace id: 12345parent: rootnode 1
Span B trace id: 12345parent: Anode 2
Span C Span Dtrace id: 12345parent: B
trace id: 12345parent: Bnode 3
RPC
RPC RPC
4 Copyright © 2014 NTT DATA Corporation
Starting root span
FsShell shell = new FsShell();! conf.setQuietMode(false);! shell.setConf(conf);! int res = 0;! SpanReceiverHost.getInstance(new HdfsConfiguration());! TraceScope ts = null;! try {! ts = Trace.startSpan("FsShell", Sampler.ALWAYS);! res = ToolRunner.run(shell, argv);! } finally {! shell.close();! if (ts != null) ts.close();! }
org.htrace.Trace#startSpan create spannew trace ID is assigned when root span starts
5 Copyright © 2014 NTT DATA Corporation
Starting passive span
Starting child span only when there is parentFor server-‐‑‒side custom tracing span
if (Trace.isTracing()) {! traceScope = Trace.startSpan(method.getName());! }
6 Copyright © 2014 NTT DATA Corporation
Passing tracing info along with RPC
message RequestHeader {! optional uint32 call_id = 1;! optional RPCTInfo trace_info = 2;! optional string method_name = 3;! ...!}!!message RPCTInfo {! optional int64 trace_id = 1;! optional int64 parent_id = 2;!}!
RPC header has optional field for tracingRPC with tracing info start span on server-‐‑‒side
7 Copyright © 2014 NTT DATA Corporation
Span receivers
Each process loads receiver moduleReceivers receive spans from in-‐‑‒process queue Receivers send spans to collector asynchronously
Server
Client
Server
Collector/Sink
SpanReceiver
SpanReceiver
SpanReceiver
RPCRPC
Tracing Spans
8 Copyright © 2014 NTT DATA Corporation
Passing tracing info between threads
Ongoing tracing span is stored in ThreadLocalYou need to pass tracing info between threadsif (header.hasTraceInfo()) {! // If the incoming RPC included tracing info, always continue the trace !
TraceInfo parentSpan = new TraceInfo(header.getTraceInfo().getTraceId(),! header.getTraceInfo().getParentId());! traceSpan = Trace.startSpan(rpcRequest.toString(), parentSpan).detach();!}!Call call = new Call(header.getCallId(), header.getRetryCount(),! rpcRequest, this, ProtoUtil.convert(header.getRpcKind()),! header.getClientId().toByteArray(), traceSpan);!!...!!if (call.traceSpan != null) {! traceScope = Trace.continueSpan(call.traceSpan);!}!
9 Copyright © 2014 NTT DATA Corporation
JIRAs
Already available in HBase (HBASE-‐‑‒6449) HDFS (HDFS-‐‑‒5274)
Working on YARN (YARN-‐‑‒1418)
10 Copyright © 2014 NTT DATA Corporation
Configurations
Setting receiver class impl turns on tracingEach receiver impl has its own additional confs! <property>! <name>hbase.trace.spanreceiver.classes</name>! <value>org.htrace.impl.HBaseSpanReceiver</value>! </property> ! <property>! <name>hbase.htrace.hbase.collector-quorum</name>! <value>127.0.0.1</value>! </property>
11 Copyright © 2014 NTT DATA Corporation
Tracing from HBase shell
"trace" command start/stop tracing span# You need configuration on client node!
$ hbase shell! > trace 'start'! > create 'test', 'f'! > trace 'stop'
16 Copyright © 2014 NTT DATA Corporation
Todo
Adding granular tracing spansSampling and filtering spansDynamic reconfiguration (HDFS-‐‑‒6956)sink and viewer with less dependency