20140329 modern logging and data analysis pattern on .NET
-
Upload
takayoshi-tanaka -
Category
Technology
-
view
1.489 -
download
5
description
Transcript of 20140329 modern logging and data analysis pattern on .NET
![Page 1: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/1.jpg)
![Page 2: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/2.jpg)
![Page 3: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/3.jpg)
仕事
on AWS
個人活動
http://tech.tanaka733.net http://tech.en.tanaka733.net
@tanaka_733
![Page 5: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/5.jpg)
![Page 6: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/6.jpg)
ログ収集の目的
![Page 7: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/7.jpg)
異常検知
障害発生時の調査材料
パフォーマンス改善
ビジネス上の目的が達成されたかの判断材料
![Page 8: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/8.jpg)
Windows環境およびGraniにおける現状
![Page 9: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/9.jpg)
普通 ASP.NET MVC 5
1万
1億
100台
![Page 10: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/10.jpg)
using
![Page 11: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/11.jpg)
![Page 12: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/12.jpg)
![Page 13: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/13.jpg)
![Page 14: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/14.jpg)
ウェブサーバー
![Page 15: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/15.jpg)
![Page 16: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/16.jpg)
Windowsサポート
強力なクエリ
![Page 17: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/17.jpg)
http://www.sumologic.com/product/solutions/application-management/
![Page 18: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/18.jpg)
![Page 19: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/19.jpg)
何か解はないのか…
![Page 20: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/20.jpg)
Semantic Logging
![Page 21: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/21.jpg)
![Page 22: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/22.jpg)
![Page 23: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/23.jpg)
![Page 24: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/24.jpg)
ETW
EventSource
クラスSLAB
![Page 25: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/25.jpg)
http://msdn.microsoft.com/ja-jp/magazine/cc163437.aspx
![Page 26: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/26.jpg)
![Page 27: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/27.jpg)
![Page 28: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/28.jpg)
![Page 29: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/29.jpg)
ライブラリの追加なし
![Page 30: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/30.jpg)
![Page 31: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/31.jpg)
Event Sink
アプリケーションEventSourceを継承した自前のイベントソース
ObservableEventListener Event Sink
http://msdn.microsoft.com/en-us/library/dn440729(v=pandp.60).aspx#sec2
![Page 32: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/32.jpg)
http://msdn.microsoft.com/en-us/library/dn440729(v=pandp.60).aspx#sec2
Event Sink
アプリケーションEventSourceを継承した自前のイベントソース
Event Sink
ETW
Trace Event Service
アプリ本体
ロギングアプリ
![Page 33: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/33.jpg)
![Page 34: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/34.jpg)
[EventSource(Name = "Grani")]public class GraniEventSource : EventSource{
private readonly static GraniEventSource log = new GraniEventSource();
private GraniEventSource() { }
public static GraniEventSource Log { get { return log; } }
[Event(1, Message = "Application Failure: {0}",Level = EventLevel.Critical, Keywords = Keywords.Diagnostic)]internal void Failure(string message){
WriteEvent(1, message);}
}
![Page 35: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/35.jpg)
![Page 36: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/36.jpg)
http://blogs.msdn.com/b/agile/archive/2014/03/27/semantic-logging-application-block-now-supports-
activity-tracing-and-elasticsearch.aspx
![Page 38: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/38.jpg)
public class S3Sink : IObserver<EventEntry>, IDisposable{
public void OnNext(EventEntry value){
//ログ1件を処理する}
}
EnterpriseLibrary.SemanticLogging
BufferedEventPublisher<EventEntry>
![Page 39: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/39.jpg)
<?xml version="1.0" encoding="utf-8" ?><configuration >
<traceEventService/><sinks>
<consoleSink name="console" ><sources>
<eventSource name="Grani" level="LogAlways"/></sources>
</consoleSink><customSink name="s3redissink" type ="Grani.EventCollector.S3Sink.S3Sink, Grani.EventCollector.S3Sink">
<sources><eventSource name="Grani" level="LogAlways"/>
</sources><parameters>
<parameter name="bucketName" type="System.String" value="green-eventtrace-logs" /></parameters>
</customSink></sinks>
</configuration>
![Page 40: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/40.jpg)
Fluentd on Linux
![Page 41: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/41.jpg)
![Page 42: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/42.jpg)
![Page 43: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/43.jpg)
![Page 44: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/44.jpg)
![Page 45: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/45.jpg)
SLAB による
分析基盤の構築
![Page 46: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/46.jpg)
HDInsight (Hadoop)Storage BLOB
![Page 47: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/47.jpg)
Amazon EC2Amazon S3 Amazon Redshift
AWS Data Pipeline
![Page 48: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/48.jpg)
![Page 49: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/49.jpg)
![Page 50: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/50.jpg)
分析ツールの使い分け
![Page 52: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/52.jpg)
![Page 53: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/53.jpg)
![Page 54: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/54.jpg)
await
![Page 55: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/55.jpg)
http://channel9.msdn.com/posts/Introducing-Semantic-Logging
http://channel9.msdn.com/Events/Build/2013/3-336
![Page 56: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/56.jpg)
http://msdn.microsoft.com/en-us/library/dn440729(v=pandp.60).aspx
http://slab.codeplex.com/
http://blogs.msdn.com/b/agile/archive/2014/03/27/semantic-logging-application-block-now-supports-activity-tracing-and-elasticsearch.aspx
![Page 57: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/57.jpg)
http://msdn.microsoft.com/ja-jp/magazine/cc163437.aspx
http://blogs.msdn.com/b/dotnet/archive/2014/01/30/microsoft-diagnostics-tracing-eventsource-rtms.aspx?Redirected=true
![Page 58: 20140329 modern logging and data analysis pattern on .NET](https://reader033.fdocuments.us/reader033/viewer/2022051609/5478ec0eb4795972098b4654/html5/thumbnails/58.jpg)