Introducing log analysis to your organization

IntroducingLogAnalysisToYourOrganization

RafałKuć

Sematext UndMich

metrics

cloud&

Next60minutes…

Logshipping- buffers- protocols- parsing

Centralbuffering- Kafka- Redis

Storage&Analysis- Elasticsearch- Kibana- Grafana

Why&How?- ShouldItry?- Opensource- Commercial

WhyYouShouldCare

Environmentsaregettingbigger

WhyYouShouldCare

Containersareeverywhere

WhyYouShouldCare

Infrastructureworkgetsautomated

CreatedbyKjpargeter - Freepik.com

WhyYouShouldCare

Logs&metricsatthesameplace

WhyYouShouldCare

Fasterdiagnostics==lessmoneyspent

Logs&metricsatthesameplace

GoingForCommercialSolution

IconmadebySmashicons from www.flaticon.com

GoingOpen-Source

GoingOpen-Source– Today’sFocus

Logshippingarchitecture

File Shipper

CentralizedBuffer

File Shipper

CentralizedBuffer

File Shipper

CentralizedBuffer

ES ES ES

Focus:Shipper

File Shipper

CentralizedBuffer

ES ES ES

Whatabouttheshipper?

CentralizedBuffer

Whichshippertouse?

Whichprotocol shouldbeused

Whataboutthebuffering

LogtoJSON orparse andhow

Buffers

performance & availability

batches&threads whencentralbufferisgone

Buffertypes

Disk ||memory ||combinedhybrid approachOnsource||centralized

Buffer

fileorlocallogshipper

easyscaling– fewermovingpartsoftenwiththeuseoflightweightshipper

Kafka /Redis /Logstash /etc…

oneplaceforallchangesextrafeaturesmadeeasy(likeTTL)

BuffersSummary

Simple Reliable

Buffer

Protocols

UDP– fast,coolfortheapplication,notreliableTCP – reliable(almost) applicationgetsACK whenwritten tobuffer

Application levelACKsmaybeneeded

Logstash,rsyslog,Fluentd

Logstash,rsyslog

Logstash,Filebeat

Logstash,rsyslog,Filebeat,Fluentd

Choosingtheshipper

application

rsyslog Elasticsearchhttp

socket

memory&diskassistedqueues

FinalArchitecture

application

socket

application

filersyslogLogagentfilebeat

consumer

FinalArchitecture

application

socket

application

rsyslogLogagentfilebeat

consumer

ParsingDoneHere

Focus:CentralizedBuffer

File Shipper

CentralizedBuffer

ES ES ES

WhyApacheKafka?

Fast &easytouse

Easytoscale

Faulttolerantandhighlyavailable

Supportsstreaming

Worksinpublish/subscribemode

Kafkaarchitecture

ZooKeeper

KafkaKafka

Kafka&topics

security_logs access_logs

app1_logs app2_logs

Kafkastoresdatain topics

writtenondisk

Kafka&topics&partitions&replicas

logspartition2

logspartition1

logspartition3

logspartition4

logsreplicapartition2

ScalingKafka

logspartition1

ScalingKafka

logspartition1

logspartition2

logspartition3

logspartition4

ScalingKafka

logspartition1

logspartition2

logspartition3

logspartition4

logspartition5

logspartition6

logspartition7

logspartition8

logspartition9

logspartition10

logspartition11

logspartition12

logspartition13

logspartition14

logspartition15

logspartition16

ThingstorememberwhenusingKafka

Scales byaddingmorepartitions notthreads

ThemoreIOPS thebetter

Keepthe#ofconsumersequalto#ofpartitions

Replicas usedforHA andFT only

Offsets storedperconsumer– multipledestinationseasilypossible

Focus:Elasticsearch

File Shipper

CentralizedBuffer

ES ES ES

Elasticsearchclusterarchitecture

client

master

ingest

Dedicatedmastersplease

client

master

discovery.zen.minimum_master_nodes ->N/2+1mastereligiblenodes

ingest

Elasticsearch– Indices

Index – logicalplacefordata

Index– canbecomparedtodatabaseinDB

Index– builtoutofoneormoreshards

Shard – canbespreadamongmultiplenodes

ScalingElasticsearch

LogsShard1

UsersShard1

InvoicesShard1

LogsShard1

LogsShard2

LogsShard3

LogsShard4

LogsShard3

LogsShard2

LogsShard4

LogsShard1

LogsReplica4

LogsShard2

LogsReplica3

LogsShard4

LogsReplica1

LogsShard3

LogsReplica2

Onebigindexisano-go

Notscalableenoughfortimebaseddata

Onebigindexisano-go

Indexingslowsdownwithtime

Onebigindexisano-go

Expensivemerges

Onebigindexisano-go

Expensivemerges

Delete byquery neededfordataretention

Dailyindicesareagoodstart

2017.11.16 2017.11.17 2017.11.20 2017.11.21...

Indexing isfaster forsmallerindices

Deletes arecheap

Search canbeperformedonindicesthatareneeded

Static indicesarecachefriendly

indexing

mostsearches

Dailyindicesareagoodstart

2017.11.16 2017.11.17 2017.11.20 2017.11.21...

Indexing isfaster forsmallerindices

Deletes arecheap

Search canbeperformedonindicesthatareneeded

Static indicesarecachefriendly

indexing

mostsearches

Wedelete wholeindices

Dailyindicesaresub-optimal

friday

saturdaysunday

loadisnoteven

Sizebasedindicesareoptimal

sizelimitforindices

logs_01

indexing

around5– 10GBpershardonAWS

sizelimitforindices

logs_01

indexing

sizelimitforindices

logs_01

indexing

logs_02

sizelimitforindices

logs_01

indexing

logs_02

sizelimitforindices

logs_01 logs_02

indexing

logs_N...

Sliceusingsize

Predictable searchingandindexingperformance

Better indicesbalancing

Fewershards

Easier handling ofspikyloads

Lesscostsbecauseofbetter hardwareutilization

ProperElasticsearchconfiguration

Keepindex.refresh_interval atmaximumpossiblevalue1sec->100%,5sec->125%,30sec-> 175%

Youcanloosen upmerges- possiblebecauseofheavyaggregationuse- segments_per_tier ->higher-max_merge_at_once->higher-max_merged_segment ->lower

Allprefixedwithindex.merge.policy

} higherindexingthroughput

ProperElasticsearchconfiguration

Index onlyneededfields

Usedocvalues

Donotindex_source

Donotstore_all

Optimizationtime

Wecanoptimize datanodesfortimebaseddata

client

master

ingest

Hot– coldarchitecture

EShot EScold EScold

-Dnode.attr.tag=hot -Dnode.attr.tag=cold -Dnode.attr.tag=cold

logs_2017.11.22

EShot EScold EScold

-Dnode.attr.tag=hot -Dnode.attr.tag=cold -Dnode.attr.tag=cold

curl-XPUTlocalhost:9200/logs_2017.11.22 -d'{"settings":{"index.routing.allocation.exclude.tag":"cold","index.routing.allocation.include.tag":"hot"}}'

logs_2017.11.22

EShot EScold EScold

indexing

logs_2017.11.22logs_2017.11.23

EShot EScold EScold

indexing

logs_2017.11.22logs_2017.11.23

EShot EScold EScold

indexing

moveindexafterdayends

curl-XPUTlocalhost:9200/logs_2017.11.22/_settings-d'{"index.routing.allocation.exclude.tag":"hot","index.routing.allocation.include.tag”:"cold"

logs_2017.11.23 logs_2017.11.22

EShot EScold EScold

indexing

logs_2017.11.23logs_2017.11.24 logs_2017.11.22

EShot EScold EScold

indexing

logs_2017.11.23logs_2017.11.24 logs_2017.11.22

EShot EScold EScold

indexing

moveindexafterdayends

logs_2017.11.24 logs_2017.11.22 logs_2017.11.23

EShot EScold EScold

indexing

HotESTier

GoodCPULotsofI/O

ColdESTier

MemoryboundDecentI/O

EScold

ColdESTier

MemoryboundDecentI/O

Hot– coldarchitecturesummary

EScold

Optimizecosts – differenthardwarefordifferenttier

Performance – usecaseoptimizedhardware

Isolation – longrunningsearchesdon’taffectindexing

Elasticsearchclient nodeneeds

client

master

ingest

Elasticsearchclient nodeneeds

Nodata=noIOPS

Largequerythroughput=highCPUusage

Lotsofresults=highmemory usage

Lotsofconcurrentqueries=higherresources utilization

Elasticsearchingest nodeneeds

client

master

ingest

Elasticsearchingestnodeneeds

Nodata=noIOPS

Largeindexthroughput=highCPU&memoryusage

Complicatedrules=highCPUusage

Largerdocuments=moreresources utilization

Elasticsearchmaster nodeneeds

client

master

ingest

Elasticsearchingestnodeneeds

Nodata=noIOPS

Largenumberofindices=highCPU&memoryusage

Complicatedmappings=highmemoryusage

Dailyindices=spikesinresources utilization

WhataboutOS?

SayNO toswapSettherightdiskscheduler

CFQ forspinningdisksdeadline forSSD

Usepropermount optionsforext4noatimenodirtimedata=writeback,nobarier

ForbaremetalcheckCPUgovernordisabletransparenthugepages

/proc/sys/vm/nr_hugepages=0

Analysis- Kibana

Analysis- Grafana

WhereToGoFromHere?

Weareengineers!

Wedevelop DevOpstools!

WeareDevOps people!

Wedofunstuff;)http://sematext.com/jobs

Thankyouforlistening!Getintouch!

Rafałrafal.kuc@sematext.com@kucrafal

http://sematext.com@sematext http://sematext.com/jobs

Introducing log analysis to your organization

Engineering

Transcript of Introducing log analysis to your organization

Happy Wednesday! Tests are not graded New hw stamp log today Binder organization.

REQUEST FOR PROPOSAL Hiring firm/organization Introducing ...

Simple strategies for introducing lean in your workplace... · INTRODUCING LEAN IN YOUR WORKPLACE PRESENTED BY: ... organization with baselines and targets. Using the template as

Introducing the founder of Entrepreneurs’ Organization ...

IRJET-IMPORTANCE OF CENTRALIZED LOG SERVER AND LOG ANALYZER SOFTWARE FOR AN ORGANIZATION

Organization at the Leading Edge: Introducing Holacracy™

Women’s Rights Organization, Introducing NOW

Large-Scale Log Management Deployment...• introducing new plugins, • building log parsers, • data processing pipelines, • managing alerting and notiﬁcations, • including

DOs and DON Ts of introducing AGILE to hardware/industrial ......of introducing AGILE to hardware/industrial organization ... development teams •Low usage of products/appliances

APPENDIX A EIS Maritime Industry Contact Interview Log and Interview … · 2011-05-23 · EIS Maritime Industry Contact Interview Log and Interview Summaries. Organization Significance

Language Guide and Reference - IBMpublib.boulder.ibm.com/tividd/td/TDS390/SH19-6817-07/en... · 2004-02-16 · Introducing the log collector 3 Collecting log data .....3 Listing log

Introducing the KCRI ePhD Log

Introducing the Conservation GeoPortal - OAS - Organization of

Generex Technology Co.,Ltd. Agenda Company Profile Generex Organization Chart Introducing Products Our Customer.

Extraordinary Leadership Introducing Strategic Thinking To The Organization

Welcome!!! CDSS, DCFS, Probation Title IV E Waiver Learning Organization Group (LOG)

The jujutsu of introducing usability to an organization

Introducing the Jahia Log Analyzer

Organization at the Leading Edge: Introducing Holacracy™library.uniteddiversity.coop/Decision_Making_and...Organization at the Leading Edge: Introducing Holacracy By Brian J. Robertson

Estate Planning: Personal Log · Estate Planning: Personal Log. Provided by Navy Mutual . Nonprofit, Veterans Service Organization Since 1879. 800-628-6011 / customerservice@navymutual.org