Logger reporting, the whole reporting and nothing but … – describes which events to include in...
Transcript of Logger reporting, the whole reporting and nothing but … – describes which events to include in...
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Logger reporting, the whole reporting and nothing but the reporting Johnny Khoury, CISSP Global Field Engineer Aaron Kramer, CISSP, CEH, Global Field Engineer, @SecGeek #HPProtect
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 2
Agenda • Define report, query, parameter (Aaron) • Managing report jobs (Johnny) • Compare tuned and non-tuned reports (Johnny) • Good vs. bad SQL queries (Johnny) • Sharing reports: Export, import (Aaron) • Running report using Logger API (Aaron)
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Define report, query, parameter
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 4
Define report, query, parameter
Reporting provides repeatable, schedulable, summarization and detail of events • Logger report
– Consists of a query and a time range over which to run that query – Can be run on demand via UI, on a schedule, or over the Logger API – Output formats include HTML, PDF, MS Excel, CSV, MS Word, Interactive HTML, XML – Use Logger “Report Explorer” to view, copy, modify, run
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 5
Define report, query, parameter
Query – describes which events to include in the report, and how (if) to summarize • Logger query
– Can be created using GUI or writing SQL directly – SQL = Structured Query Language – More than 130 ship with Logger, can copy, modify – Refer to MySQL 5.1 syntax as a guide – Use Logger “Query Explorer” to view, copy, modify, test
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 6
Define report, query, parameter
Query – describes which events to include in the report, and how (if) to summarize
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 7
Define report, query, parameter
Parameter – optional part of a query, a ‘substitute-able’ field • Logger parameter
– Want to include the sourceUserName in a report, but the value you want could be anyone, not known in advance
– Create a UserName parameter, define length, type of data – Use the parameter in the query, which is used in the report
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Managing report jobs
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 9
Managing report jobs
• Avoid scheduling too many reports to run at the same time • Schedule reports during non peak hours • Leverage dynamic time frame for running report. No need to schedule all reports to run
“00:00:01 am” for the previous day • For example schedule report to run at 2 am for the previous day • Start ( $NOW -26h) End ($Now -2h)
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 10
Report Jobs Execution Status: (job summary)
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Tuned vs. non-tuned reports Good vs bad SQL queries
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 12
Logger report and general performance
The performance of a report SQL query will be affected by a combination of the following factors:
• Data volume – the larger the data volume, the more data needs to be processed • Data distribution – this includes uniqueness • Server load : This includes the number of searches and reports running concurrently • Query complexity – queries can be crafted to be more optimal. This will differ on a case-by-case basis
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 13
Brief background about Logger reports and UI search
Similarities: Both Logger UI Search and Logger MySQL run search in the backend to extract the data Both rely on Logger MySQL plugin to extract data from the storage Engine Both Logger Report and UI Search condition are translated into SQL-like condition (with the exception of
regex based search)
Select events.arc_name , events.arc_sourceAddress from events where name like ‘Failed%’ group by events.arc_name , events.arc_sourceAddress; vs
name contains "Failed*" | chart count by name sourceAddress Key difference: Logger report use sql query functions ( GROUP, ORDER, LIMIT ) aggregate and sort Logger data Logger UI Search use search command (TOP, CHART, SORT, DEDUP) to sort and aggregate the results
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 14
MySQL workflow
MySQL -storage engine workflow
* The JDBC connection is not open. Each search is giving search session ID * The results of search/sql output is stored in Temp table
Select …
from arc_event Where …
Order by …
Logger server Logger
MySQL plugin
Request for Metadata
Metadata
rows
JDBC for SQL request
(session id)
MySQL
Storage engine
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 15
1) Use Indexable fields in the SQL Query
Why are my reports are slow ? If any of the fields selected in the sql query are not indexed, the report speed will default to the slow non-index search. (By default 60 fields are indexed, they are listed under the search option) Bad query select events.arc_deviceHostName, events.arc_requestUrlQuery, events.arc_requestMethod, events.arc_userName, events.arc_CategoryOutcome, from events where events.arc_sourceAddress="10.10.10.12";
( Note events.arc_requestUrlQuery and events.arc_requestMethod are not indexed by default)
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 16
2) Leverage the super indexed fields in the WHERE CONDITION for the SQL
SuperIndex the report – With the introduction of Super-Index fields (Bloom Filters) in Logger 5.5 , there are 15 default fields that
users can select to improve the search speed significantly – The superindex fields need to be entered in the WHERE condition
deviceEventClassId deviceProduct deviceVendor destinationHostName
destinationPort destinationAddress destinationUserId destinationUserName
deviceAddress deviceHostName sourceHostName sourcePort
sourceAddress sourceUserId sourceUserName
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 17
Continue – leverage super-index fields
Example: no Super Index fields in the WHERE Condition select events.arc_sourceAddress, events.arc_sourceUserName, events.arc_requestMethod, SUM(events.arc_baseEventCount) , HOUR(events.arc_endTime) as "Hour“ FROM events WHERE events.arc_requestMethod = "GET“ AND events.arc_applicationProtocol CONTAINS "http“ GROUP BY events.arc_sourceAddress, events.arc_sourceUserName, events.arc_requestMethod, Hour ORDER BY HOUR
Good Query: Faster query using Super-Index fields in the WHERE condition select events.arc_sourceAddress, events.arc_sourceUserName, events.arc_requestMethod, SUM(events.arc_baseEventCount) ,HOUR(events.arc_endTime) as "Hour” FROM events WHERE events.arc_deviceVendor="Apache“ AND events.arc_deviceHostName = "buildserver“ GROUP BY events.arc_sourceAddress, events.arc_sourceUserName, events.arc_requestMethod, Hour ORDER BY HOUR
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 18
3) Use Limit option where possible
Another time and resource saver that can improve the SELECT statement to run faster and consume less memory cache is the LIMIT option. Example - Top 10 source Address select events.arc_sourceAddress, events.arc_sourceUserName, SUM(events.arc_baseEventCount) as "COUNT“ FROM events WHERE events.arc_deviceVendor="Apache" AND events.arc_sourceAddress IS NOT NULL GROUP BY events.arc_sourceAddress, events.arc_sourceUserName ORDER BY COUNT desc limit 10;
( BY default order will sort is asc order)
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 19
4) Avoid using sub-query if possible (slow)
– Sub queries are not handled well in MySQL by default. – The Sub-query can consume more space in memory and disk.( 2 temp tables to scan) Bad query select arc_eventTime, arc_name, arc_sourceAddress, arc_sourceHostName, arc_sourceProcessName, arc_sourceUserName, arc_destinationUserId, arc_destinationUserName from events where events.arc_sourceAddress in ( select distinct(events.arc_sourceAddress) from events where events.arc_sourceAddress != '192.168.36.34' AND events.arc_sourceAddress != ‘192.168.35.23’ AND events.arc_sourceAddress != '192.168.37.34’)
Good query select arc_eventTime, arc_name, arc_sourceAddress, arc_sourceHostName, arc_sourceProcessName, arc_sourceUserName, arc_destinationUserId, arc_destinationUserName from events where events.arc_sourceAddress != '192.168.36.34' AND events.arc_sourceAddress != 192.168.35.23 AND events.arc_sourceAddress != '192.168.37.34
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 20
5) GROUP BY with order by & GROUP BY... HAVING
GROUP BY is an aggregate function that operate on set values. It is equivalent to group on all rows based on selected fields. There are couple options to utilize GROUP BY : a) GROUP BY optimization with ORDER BY. select events.arc_sourceAddress, events.arc_destinationUserId, events.arc_categoryOutcome, SUM(events.arc_baseEventCount) as "COUNT", HOUR(events.arc_endTime) as "HOUR" FROM events WHERE events.arc_deviceVendor="Apache" AND events.arc_deviceHostName = "buildserver.dev" GROUP BY events.arc_sourceAddress, events.arc_destinationUserId, events.arc_categoryOutcome ORDER BY HOUR
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 21
Continue GROUP BY (1 page vs. 15 page)
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 22
Continue: Group BY …. HAVING
b) GROUP BY ..... HAVING Adding filter condition to aggregation for summary results. Report Failed logins attempt more than 3 time select events.arc_destinationUserName, COUNT(arc_baseEventCount) as "COUNT“ from events where events.arc_categoryBehavior="/Authentication/Verify" and events.arc_categoryOutcome="/Failure" Group by events.arc_destinationUserName Having COUNT > 3;
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 23
5) Use few columns where possible. Size matters
In the MySQL world, size is everything Avoid including too many fields in the SELECT statement. Use types that are smaller in size if possible. DestinationZoneUri=/All Zones/ArcSight System/Private Address Space Zones/RFC1918: 10.0.0.0-10.255.255.255 . The size of this value =89 bytes. msg =Network: A user or computer logged on to this computer from the network. (73 bytes) name=An account failed to log on. ( 29 bytes)
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 24
CONTINUE
Too many fields in this query: SELECT events.arc_deviceVendor, events.arc_deviceProduct ,events.arc_destinationZoneURI ,events.arc_name,
events.arc_destinationAddress , events.arc_destinationHostName, events.arc_destinationUserName, events.arc_sourceUserName ,events.arc_categoryOutcome , events.arc_endTime FROM EVENTS WHERE (events.arc_deviceVendor = "Microsoft" AND events.arc_deviceEventClassId = "Security:540")
order by events.arc_destinationUserName
For faster query: We can remove events.arc_deviceVendor,events.arc_destinationZoneURI , events.arc_deviceProduct ). SELECT ,events.arc_name, events.arc_destinationAddress , events.arc_destinationHostName, events.arc_destinationUserName, events.arc_sourceUserName , events.arc_categoryOutcome , events.arc_endTime FROM EVENTS WHERE (events.arc_deviceVendor = "Microsoft" AND events.arc_deviceEventClassId = "Security:540")
order by events.arc_destinationUserName
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 25
8) Use NOT NULL if you can
NULL columns require additional space in the row to record whether their values are NULL. Some of the device hostnames do not populate with a hostname, so it can throw the device counts off.
Select …. WHERE events.arc_deviceHostName IS NOT NULL GROUP BY events.arc_deviceHostname
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Sharing Logger reports
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 27
Sharing Logger reports
Sharing reports increases in value exponentially* (not a financial commitment) • Logger lets you bundle up parameters, queries, and reports, and export them to a file • Logger lets you import report bundle(s) from others • These bundles are called CAB files (file extension is .CAB), along with a resource file called a .CONF file • For the security-minded, no digital signing exists, no risk assessment is implied • Other option would be to post the actual query used, and a screen shot of the report
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 28
Sharing Logger reports
Logger export • Encouraged to also post the actual query used, and a screen shot of the report • Reporting, Administration, iPackager
– “import” items from the Logger Report Server – build the .CAB file
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 29
Sharing Logger reports
Logger import • See if someone else has written the report you want, or something like it • Look at their screen shot, or query • Copy n paste OR obtain their .CAB file
– Many times copy and paste changes important characters such as quotes ‘ “ ` • Reporting, Administration, Deploy Report Bundle
– Upload .CAB file to Logger Server • Typical .CAB file is small: from 10KBytes to 100KBytes to 200 KBytes
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Run reports with Logger API
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 31
Run reports with Logger API
Logger API is a programmatic way to run Logger reports (and Logger search, too!) • Logger API is fully documented in “Logger_WebServicesAPI_5.5.pdf”
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 32
Resources
Additional resources • Logger 5.5 Administrators Guide (PDF) https://protect724.hp.com/community/arcsight/productdocs/logger • My SQL 5.5 Reference: http://dev.mysql.com/doc/refman/5.5/en/functions.html • Logger API PERL code: https://protect724.hp.com/message/24390#24390
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 33
For more information
Attend these sessions
• TT3132, Army of loggers: : Searching for that needle at billions of EPS
• TT3154, : Renovation of the HP ArcSight Logger user interface
• TT3589, Enriching HP ArcSight Logger search with third-party information
Visit these demos
DEMO 3526 – HP ArcSight Logger
After the event
Contact your sales rep
Visit http://protect724.hp.com
Your feedback is important to us. Please take a few minutes to complete the session survey.
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 34
Please fill out a survey. Hand it to the door monitor on your way out.
Thank you for providing your feedback, which helps us enhance content for future events.
Session TB3123 Speaker Johnny Khoury, Aaron Kramer
Please give me your feedback
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.