SplunkLive 2011 Advanced Session

May 17, 2011

Jeff Blake

Technical WorkshopsAdvanced User Training

Copyright © 2011, Splunk Inc. Listen to your data.2

Agenda

AssumptionsBuilding AppsUsers and RolesSplunk By Example – Deconstructing the DemoSupport and Community

Assumptions


You Are in This Session Because….

• You have developed advanced searches with Splunk to manipulate and present data

• You have mastered sourcetyping and extracting fields• You have built reports beyond | timechart count• You have created dashboards of some kind• You have http://www.splunk.com/base/Documentation bookmarked• You have seen all of the Splunk Ninja videos

http://www.splunk.com/base/Documentation


Building Apps


What is An App, Anyway?

• Terminology• Apps – A workspace that solves a specific use case with a navigable view• Add-on – A reusable Splunk component that does not contain a view

• Example• Splunk for Cisco Security is an App• The collection of field extractions/sourcetypes/transforms/eventypes that

map raw firewall logs is an Add-on


What You Need to Create an App

• Some data • Add-ons (Use ours or build your own)• Splunk Objects

• Saved Searches• Reports• Dashboards• Views

• Text Editor (Komodo Edit, Text Wrangler, vi, Notepad) for Advanced XML• Web development tools (Firebug, etc)


Getting Started is Easy

This will build the necessary directory structure in$SPLUNK_HOME/etc/apps


app.confThe Interesting Stuff:• version = <version string> - Give your App a version number. Important if

you plan to put on Splunkbase• id = <appid> - Must be same as folder name where your app lives. With version,

used to check for App updates on Splunkbase• state_change_requires_restart = true | false - Install Apps without

restarting! Check $SPLUNK_HOME/etc/system/default/apps.conf. If Splunk touches any endpoint not listed, restart required

• build = <integer> - Change with version so browsers don’t use cached copies of old static files

Users and Roles


Users and Roles

Problem Investigation Problem Investigation Problem Investigation

Save Searches

Share Searches

LDAP, AD Users and Groups

Splunk Flexible Roles

Manage Users

Manage Indexes

Capabilities & Filters

NOT tag=PCI

App=ERP…

Map LDAP & AD groups to flexible Splunk roles. Define any search as a filter.

Integrate authentication with LDAP and Active Directory.


Maps Roles to AppsOptions for Logical Data Partitioning• Map Roles to Indexes • Map Roles to Sourcetypes

Once your Apps are built, mapping Apps to Roles is easy:


Other User Control Methods

Edit Permissions by App in the App Manager

Remove options for users in the AccountBar

<module name="AccountBar" layoutPanel="appHeader"> <param name="mode">lite</param></module>

Splunk By ExampleBasic UI Customization


*Disclaimer

The App and Splunk Web customizations that we will show in this presentation are by no means comprehensivePlease see the Developer Manual for more detail

http://www.splunk.com/base/Documentation/latest/Developer/


Launch Page

In the demo we saw a slide as a landing page instead of a dashboard.How did you do that?

Screenshot here


Where Stuff Lives

$SPLUNK_HOME/etc/apps/your_app_name$SPLUNK_HOME/etc/apps/your_app_name/default– Put all Splunk configuration files for you App here

$SPLUNK_HOME/etc/apps/your_app_name/default/data/ui/nav– Contains default.xml, defines navigation menus

$SPLUNK_HOME/etc/apps/your_app_name/default/data/ui/views– All dashboard and view xml files live here

$SPLUNK_HOME/etc/apps/your_app_name/default/appserver– Add images, CSS, or HTML


Where Stuff Lives, Cont.

$SPLUNK_HOME/etc/apps/your_app_name/bin– Custom scripts or executables for your App

$SPLUNK_HOME/etc/apps/your_app_name/local– For users and admins to edit default configurations

$SPLUNK_HOME/etc/apps/your_app_name/metadata– Stores object permissions


Working Backward - Navsdefault/data/ui/nav/default.xml

<nav>

<collection label="About">

<view name="arch" default="true" />

<divider />

<a href="mailto:[email protected]">Send Feedback...</a>

</collection>

<collection label="IT Operations">

<view name="operational_visibility" />

<view name="env_state" />…


The “arch.xml” Viewdefault/data/ui/views/arch.xml

<?xml version='1.0' encoding='utf-8'?><dashboard><label>Architecture</label> <row> <html> <h1>Welcome to the Application Management Demo</h1> <h3> This Splunk instance is capturing data from a variety of applications, operating systems and network devices. </h3> <center> <a href="operational_visibility"> <img src="/static/app/appmgmt/arch.gif"/> </a> …

Link when clicked

Image to display


Manage/Edit Your Navs and Views in Splunk Web


Other Customization Options

App Icon – create your own icon to show on Home screen– $SPLUNK_HOME/etc/apps/your_app_name/appserver/static/appIcon.png

Custom Cascading Style Sheets (CSS) - background colors, fonts, logos, buttons, navigation, menus, etc– Default:

$SPLUNK_HOME/share/splunk/search_mrsparkle/exposed/css/skins/default/default.css

– Create your application.css in $SPLUNK_HOME/etc/app/your_app_name/appserver/static

– See Splunk Apps (Splunk for *NIX, Splunk for Windows, etc) for examples


Dashboards and ViewsSimplified XML– Use the Visual Dashboard editor to auto-

generate simplified XML– Adjust panel layouts– Modify underlying searches and format

charts via Report Builder or edit by hand<dashboard> <label>My dashboard</label> <row> <chart> <searchName>My saved report</searchName> <option name="charting.chart">line</option> <option name="height">200</option> </chart> </row></dashboard>


Dashboards and Views

Advanced XML– Provides extended customization and control over simplified XML– All pages and dashboards in Splunk are composed of one or many

modules– Each module contains params that control module specific

configurations

Check out available modules– http://localhost:8000/modules

View any Splunk page in Advanced XML– http://localhost:8000/en-US/app/<app_name>/<dashboard_name>?showsource=true


Operational Visibility – Current Capacity

Uses a real-time searchSets thresholds with a filler gaugeIncludes some textRotated horizontallyDisplays as a percentage


Building a Real-Time Search/Report

From the CPU sourcetype

Take the most recent event

Extract column headings to fields, only return the “pctSystem”

field

Send pctSystem to gauge command setting

ranges

2.

1.

Set time range, Real-time, 30 second

window

3.


Command Referencesourcetype=cpu | head 1 | multikv fields pctSystem | gauge pctSystem 0 25 50 75 100

Command Usage Example

head head <N> - return N events (default is 10)head <eval-expression> - a valid eval expression

| head 500| head (foo>80)

multikv multikv fields <field-list> - extracts field values from table-formatted events and filters out from extracted events fields not in the given field list

| multikv fields foo

gauge gauge <field> - transforms results into a format for display in Gauge chart types and sets gauge regions

| gauge foo 0 25 50 75 100


Building a Real-Time Search/Report

4. 5.


Edit Using Advanced XML<module name="StaticContentSample" layoutPanel="panel_row1_col3"> <param name="text">Calculated based on maximum architecture throughput and capacity.</param> </module><module name="HiddenSavedSearch" layoutPanel="panel_row1_col3" group=”Current Capacity" autoRun="True"> <param name="savedSearch">AppMgmt - Capacity - Filler Gauge</param> <param name="groupLabel">Current Capacity</param> <module name="ViewstateAdapter"> <param name="savedSearch">AppMgmt - Capacity - Filler Gauge</param> <module name="HiddenFieldPicker"> <param name="strictMode">True</param> <module name="JobProgressIndicator"> <module name="EnablePreview"> <param name="enable">True</param> <param name="display">False</param> <module name="HiddenChartFormatter"> <param name="charting.chart.usePercentageValue">true</param> <param name="charting.chart.orientation">x</param> <module name="FlashChart"> <param name="width">100%</param> <module name="ConvertToDrilldownSearch"> <module name="ViewRedirector"> <param name="viewTarget">flashtimeline</param>…..

Dashboard position

Text

Panel Title

Saved Search Name

Display as percentage

Rotate Horizontally


Add a Single Value Button

Construct your searchsourcetype="access_combined" action="purchase"

| stats avg(price) as price

| eval basketvalue = "$" + tostring(round(price,2), "commas")

| rangemap field=basketvalue severe=0-50 elevated=50-75 default=low

On our web logs, pull out the “purchase

eventsUse the “stats” command to take an average of a field

Format for display in buttonUse “rangemap” to set

thresholds


Command Referencesourcetype="access_combined" action="purchase” | stats avg(price) as price | eval basketvalue = "$" + tostring(round(price,2), "commas") | rangemap field=basketvalue severe=0-50 elevated=50-75 default=low


stats stats (stats-function(field) [as field]) + [by field-list] – provides statistics grouped optionally by field

| stats avg(foo) as foobar by bar

stats functions - avg() , c() , count() , dc() , distinct_count() , first() , last() , list() , max() , median() , min() , mode() , p<in>() , perc<int>() , per_day() , per_hour() , per_minute() , per_second() , range() , stdev() , stdevp() , sum() , sumsq() , values() , var() , varp()




eval eval eval-field=eval-expression – calculates an expression and put the result in a field

| eval foo = if(bar==200, “OK”, “Error”)

eval functions - abs(), case(), ceil() , ceiling(), cidrmatch(), coalesce(), commands(), exact(), exp(), floor(), if(), ifnull(), isbool(), isint(), isnotnull(), isnull(), isnum(), isstr(), len(), like(), ln(), log(), lower(), ltrim(), match(), max(), md5(), min(), mvappend(), mvcount(), mvindex(), mvfilter(), mvjoin(), now(), null(), nullif(), pi(), pow(), random(), relative_time(), replace(), round(), rtrim(), searchmatch(), split(), sqrt(), strftime(), strptime(), substr(), time(), tonumber(), tostring(), trim(), typeof(), upper(), urldecode(), validate()




rangemap rangemap field=<string> (<attrn>=<number>-<number>) + [default=<string>] – sets range field to the name of the ranges that match

| rangemap field=foo low=0-10 elevated=11-20 severe=21-30 default=low

Note: Splunk ships with CSS that defines colors for low, elevated, and severe. You can customize CSS for these values.


Embed Search and Time-range in View<module name="StaticContentSample" layoutPanel="panel_row1_col2"> <param name="text">Real-time average shopping cart values.</param> </module> <module name="HiddenSearch" layoutPanel="panel_row1_col2" group="RT Shopping Cart Value" autoRun="True"> <param name="search">sourcetype="access_combined" action="purchase" | stats avg(price) as price | eval basketvalue = "$" + tostring(round(price,2), "commas") | rangemap field=basketvalue severe=0-50 elevated=50-75 default=low</param> <param name="groupLabel">RT Shopping Cart Value</param> <param name="earliest">rt-1m</param> <param name="latest">rt</param> <module name="ViewstateAdapter"> <module name="HiddenFieldPicker"> <param name="strictMode">True</param> <module name="JobProgressIndicator"> <module name="SingleValue"> <param name="field">basketvalue</param> <param name="classField">range</param>.....

Our SearchReal-time 1 minute window

Use the “basketvalue” field


Build a Stacked Column Chart

Uses a lookup table to map a Product ID to a Product NameUses “timechart” command to display results over time


Use a Lookup Table

• In props.conf[access_combined]LOOKUP-prod = prod_id_lookup product_id OUTPUT product_name, price, tdf_price, call_flwrs_price

• In transforms.conf[prod_id_lookup]filename = prod_lookup.csv

• In lookup directory, prod_lookup.csvproduct_id,product_name,price,tdf_price,call_flwrs_priceRP-LI-02,Chocolate Dreams Confections,379,299,319


Construct Your Search and Format

• Either define in Report wizard • Or in view XML

<module name="HiddenChartFormatter"> <param name="chart">column</param> <param name=“stackMode”>stacked</param>


Command Referencesourcetype="access_combined” | timechart count by product_name


timechart timechart [agg=<stats-agg-term>] [<bucketing-option> ]* (<single-agg> [by <split-by-clause>] )

| timechart span=1m avg(foo) by bar

timechart functions (same as stats) - avg() , c() , count() , dc() , distinct_count() , first() , last() , list() , max() , median() , min() , mode() , p<in>() , perc<int>() , per_day() , per_hour() , per_minute() , per_second() , range() , stdev() , stdevp() , sum() , sumsq() , values() , var() , varp()


Building a Correlation Search

Uses a simple Boolean search to get results from both sourcetypesUses a search macro to process results and display


Using a Search MacroDefine in Manager > Advanced search > Search macrosReuse chunks of searches as part of your search string like a function

Invoke using the left quote character


Raw Event Output

Contain asynchronous entriesPossible duplicates


Step 1

Get rid of duplicate events or duplicate users

sourcetype=mysql_config OR sourcetype=remedy_changeticket | dedup _raw, User | transaction TicketId, User | eval hasTicket = if(eventcount > 1, "Yes", "No") | rename PrevPropValue as "Original_Value", NewPropValue as "New_Value", hasTicket as "Change_Ticket" | fields _time, User, Property, "Original_Value", "New_Value", "Change_Ticket"


Step 2

Combine two asynchronous events into one event using transaction command– Note: can also define boundaries for length of events (maxspan) or time between events

(maxpause)

Use the same “TicketId” and “User” fields between the two events to groupWill produce metadata field “duration” and “eventcount”



Step 3

Use eval to create a new field “hasTicket”Use “eventcount” field generated by transaction commandIn the results, if the uber-event has > 1 event then there is an change with an associated change ticket, otherwise “hasTicket” is “No”



Step 4

Formatting and resultsRename a few fields for clarity in dashboard resultsUse fields command to only display what we need to see


Support and Community


Support Through the Splunk Community

Browse and share Apps from Splunk, Partners and the

Community

splunkbase.splunk.com

Splunkbase

Community-driven knowledge

exchange and Q&A

answers.splunk.com

5 tracks, more than 40 sessions, the smartest Splunk users together

www.splunk.com/goto/conference


Where to Go for HelpDocumentation– http://www.splunk.com/base/Documentation

Technical Support – http://www.splunk.com/support

Videos– http://www.splunk.com/videos

Education– http://www.splunk.com/goto/education

Professional Services



http://www.splunk.com/support

http://www.splunk.com/support

http://www.splunk.com/videos

http://www.splunk.com/videos

http://www.splunk.com/goto/education

http://www.splunk.com/goto/education

Thank you

Technical WorkshopsAdvanced User Training

SplunkLive 2011 Advanced Session

Technology

Transcript of SplunkLive 2011 Advanced Session