Copyright © 2012 Splunk Inc.

Splunk All the Things:Our First 3 Months Monitoring Web Service APIs

Dan Cundiff (@pmotch) and Eric Helgeson (@nulleric)

Target Corporation

2

Agenda

Context

Problem

Solution

Examples

In progress and future stuff

Lessons and challenges

3

Context: Enterprise Services @ TargetData and transactional APIs for all the domains in our business– Products (inventory, price, description, etc)– Locations– Coupons– etc

APIs exposed inside and outsideMostly RESTful APIs, some pub sub/messagingUsed by mobile devices, applications, partners on the outside, etc.Constantly evolving, rapidly improving, all the time

4

ProblemFirst API go-live:– Millions of log events per day (grep/cut/sed/awk not cutting it)– Logs scattered everywhere– Limited access to logs– Needed end to end visibility of web services– Needed ability to discover information in logs– Can we be pro-active? Faster reactive?

Looming horizon:– BILLIONS of log events coming– Questions changing everyday from business, support, execs, developers

5

Solution: Gave Splunk a tryInstalled Splunk on a lab serverHooked up Splunk to the logsQuickly created 15+ searches and reportsGenerated a dashboard for visibility and trendingTotal time to do all this in Splunk:

~4 hours6

Why SplunkUnderstanding what’s “normal”– Identify tolerances– Identify actionable events vs. anomalies

You don’t know what you don’t know– …but Splunk can tell you what you don’t know

7

Why Splunk, part 2Indicators when are things trending badly– Proactive monitoring and recovery– Standard deviations, percentage changes over time, outliers

Full stack visibility– API gateway– Network (load balancers, firewalls)– Web/app– OS

8

Why Splunk, part 3Quick and flexible dashboardsDrill downCommunity (Splunkbase, blogs, etc)Google-able™ App store!

9

Locations Service Examples

What is “normal”?Volume

11

What is “normal”?, part 2API response time SLAs

12

What is “normal”?, part 3Errors happen, but what is acceptable?

13

404s~1700 errors once a day every week404s for stores that don’t existBot?– Who are they?– Malicious? Competitor? Individual?– Reach out to understand why

14

Understanding consumers

15

Who and how is it being used?What’s their experience?

Understanding consumers, part 2

16

Load testing in production?

Understanding infrastructureExpected design vs actual implementationNot balancing workload as expected

17

Understanding providersHow are providers responding?Is overhead added to the API response?

18

Requirements feedback loopRequirement: 200 tpsActual: ~20 tps

19

Business intelligence from APIsWhere are people searching?Where should we build our next store?How far are people traveling?What time of day?Mobile vs website?iOS vs Android?International?

20

Metrics for APIs(source: http://blog.programmableweb.com/2012/08/02/the-api-measurement-secret-know-what-metrics-matter/)

Traffic Metrics– Total calls– Top methods– Call chains– Quota faults

Developer Metrics– Total developer count– Number

of active developers– Top developers– Trending apps– Retention

Service Metrics– Performance– Availability– Error rates– Code defects

Marketing Metrics– Developer registrations– Developer portal funnel– Traffic sources– Event metrics

Support Metrics– Support tickets– Response time– Community metrics

Business Metrics– Direct revenue– Indirect revenue– Market share– Costs

21

In progress and future stuff

Splunk all the thingsConsumer appsProvider systemsOS, firewalls, proxiesExternal API gateway logsAnything in between (middleware, integrations, etc)Correlate with logs from apps degrees away (e.g. .com web logs)

Development (perf test results, git, Jenkins/CI, wiki, etc)

DashboardsGlobal dashboard summarizing all APIsBI dashboardsExecutive dashboards

24

Dashboards, part 2Environment dashboards for each API– CI– Test– Stage– Prod

25

Dashboards, part 3Alert trending dashboards for each API

26

Splunking Continuous IntegrationDrill down into CI results linked straight from Jenkins– Filtered by date OR transaction GUID

27

Splunking Continuous Integration, part 2We practice code as documentationEvery commit, Jenkins runs, extracts documentation from code, puts it in the respective wiki pages (pretty cool! – automated / no humans)Splunk monitors wiki changes using the MediaWiki APIMonitor CI + human wiki changes

https://github.com/pmotch/wikislurp

28

Common Logging ServiceCLS is our strategy for getting logs from all places into SplunkHow– Use UFs on end points everywhere– Else, consolidate and mount Splunk– Else, use CLS RESTful API

Enables end-to-end visibility– Insert GUIDs across all the hops in the transaction

Use out of the box log formats (e.g. Log4j)

29

Lessons and challenges

Lessons RTFM– Keep logs flat– Keep timestamp (ISO8601) at the beginning– k=v

Iterate quick, push to prod; minimal tweaks to SplunkFlatten out of box audit events (XML)– Toggle at runtime

Don’t re-invent the wheel, use what your system provides, Splunk can handle it!

31

Lessons, part 2 Don’t pre-optimize up front– Governance– Standards– Alerting– Access controls

Optimize as needed

32

Lessons, part 3Create a community

33

Lessons, part 4Create best practices, standards, etc in a wiki

34

Challenges: Organizational“Stop. We already have tools that do this. Use those.”– tgtMAKE saves the day– tgtMAKE = R&D– R&D = $, servers, flak shelter, people network

Make it real strategy– Demo to as many key players as possible– Drum up interested– Show actual value

35

Challenges: Organizational, part 2

36

http://knowyourmeme.com/photos/361379-shut-up-and-take-my-money

Challenges: Organizational, part 3The data can’t be trusted?

37

Challenges: OSRHEL 6SELinuxIpfwInstall notes: http://nulleric.tumblr.com/post/13855621770/splunk-on-redhat-6-install-notes

38

Challenges: InfrastructureVM requirementAdhering to MDHA requirementsUniversal Forwarder skepticism

39

Challenges: Logs on the outsideUniversal Forwarders on servers that we don’t manageFirewallsMulti-layered DMZs

40

Challenges: Splunk…

41

Challenges: Splunk (err, improvements)Index improvements– Cheap servers, can fail, can expand– Replication, N=3– Replicas on N-1 subsequent nodes– Data is always available, smooth out across servers if they go down or expand– Multi-tenant– Think OpenStack Swift “Ring” concept or Cassandra– There’s that CAP Theorem thing; they say it’s a big deal.

GUI for deployment client configurations (lazy and for n00bs, we know)Ability to extend charts with other libraries (like D3 or something)

42

Recap

Be bold. Tooling matters. Sell it.Splunk all the things!Iterate, adapt, change quickly.

43

We’re hiring(come talk to us)

44

Questions?

45