Tech Ed 2012 Real World Sharepoint Customization - Dev vs ITPro
ITPRO Log Parser and Microsoft Exchange Server the Perfect Blend Ilse Van Criekinge
63
Log Parser and Microsoft Log Parser and Microsoft Exchange Server, the Perfect Exchange Server, the Perfect Blend! Blend! Ilse Van Criekinge Exchange Trainer & Consultant (Azlan Training) [email protected] www.Pro-Exchange.be
-
Upload
razafindrabe-fanomezantsoa -
Category
Documents
-
view
124 -
download
0
Transcript of ITPRO Log Parser and Microsoft Exchange Server the Perfect Blend Ilse Van Criekinge
Log Parser and Microsoft Log Parser and Microsoft Exchange Server, the Perfect Exchange Server, the Perfect
Blend!Blend!
Ilse Van CriekingeExchange Trainer & Consultant
(Azlan Training)
ObjectivesObjectivesWho needs reports?What kind of data is available?How to gain access to the data?How to present the acquired data?
AgendaAgenda
Introduction
Process flow
Sources of data
Pulling it all together: Joins
Report creation
Automation
IntroductionIntroductionReports are critical to a healthy Exchange org
understand, monitor, and track “who-what-when-where-how” usage data
mail usage by usermessage traffic patternsdelivery timeshistorical usage trendsmessage content reporting...
Many reporting packages are available at a price
OmniAnalyser, StealthAUDIT for Exchange, eIQ Mailanalyzer, Quest MessageStats, Admin Report Kit for Exchange, IMFStats, bv-Control for Exchange, PROMODAG Reports for Exchange, MailMeter Insight, Mail Access Monitor for MS Exchange Server,MailDetective, Sirana AppAnayzer for Exchange, e-nspect real time reporting, Quest Reporter, DYS CONTROL!, Exchange Monitor, bt-LogAnalyzer, ...
But…easy to develop basic reports yourself!
RequirementsRequirementsMicrosoft Exchange ServerActive DirectoryData extraction tools
Microsoft Log Parser 2.2A dash of scripting
To create & publish reportsTo create & publish reports
Microsoft SQL Server 2000 (or better) to store the data
Visual Studio .NET 2003 (or better) to create reports
SQL Reporting Services to publish reports
Why Reporting Services?Why Reporting Services?
Powerful web based reporting toolEasy to create rich, interactive, graphical reportsEnd-users can subscribe to receive reports via email, file share, etc…End-users can export reports to various formats (XLS, XML, CSV, HTML, TIFF, PDF, etc…)Easy to developRendering and processing can be seperated
AgendaAgenda
Introduction
Process flow
Sources of data
Pulling it all together: Joins
Report creation
Automation
Process FlowProcess Flow
1. Extract data from source2. Load data into SQL Server3. Create report in Visual Studio4. Publish report
Process FlowProcess FlowExchange
Message Tracking LogsActive Directory
User & Mailbox InfoExchange
Mailbox InfoDataDataSourcesSources
DataDataAccessAccessMethodMethod
Log Parser CSVDE WMI
Microsoft SQL Server DatabaseDataDataStorageStorage
DataDataOutputOutput Microsoft SQL Server Reporting Services
AgendaAgenda
Introduction
Process flow
Sources of data
Pulling it all together: Joins
Report creation
Automation
Sources of DataSources of Data
Exchange message tracking logs
Active Directory
HomeMDB, quota settings, ...
User Information
WMI providers for Exchange
Source 1:Source 1:Exchange Message Tracking LogsExchange Message Tracking Logs
Available in Exchange 5.5/2000/2003Has to be enabled
Exchange 5.5: Information Store/MTA/Internet Mail ServiceExchange 2000/2003: Server setting
Options:Remove log files [older than (days)]Exchange 2000/2003: Enable subject logging and displayExchange 2003: Location to store log files
Message Tracking Log Message Tracking Log FormatFormat
Note: Tracking logs in Microsoft Exchange 2000 Server have a significantly different format then Microsoft Exchange Server 5.5 tracking logs.
Generally follows the W3C format for log files
First few lines contain directives, tab delimited
One log generated/server/day
Logs roll at midnight GMT
All times in the log are GMT
Message Tracking Log FieldsMessage Tracking Log FieldsField number Field name Field number Field name
1 Date 11 Priority
2 Time 12 Recipient-Report-Status
3 Client-IP 13 Total-bytes
4 Client-hostname 14 Number-recipients
5 Partner-name 15 Time-taken
6 Server-hostname 16 Encryption
7 Server-IP 17 Service-version
8 Recipient-address 18 Linked-MSGID
9 Event-ID 19 Message-subject
10 MSGID 20 Sender-address
Source: http://support.microsoft.com/?kbid=246965
Message tracking event IDs in Exchange Server 20031027: Message submission by store1028: Message deliveryOverview: http://support.microsoft.com/?kbid=821905
Message Tracking Log Message Tracking Log SampleSample
# Message Tracking Log File# Exchange System Attendant Version 6.5.7638.1
# Date Time client-ip Client-hostname Partner-Name Server-hostname server-IP Recipient-AddressEvent-ID MSGID Priority Recipient-Report-Status total-bytes Number-RecipientsOrigination-Time Encryption service-Version Linked-MSGID Message-Subject Sender-Address
2006-2-7 10:28:41 GMT - - - NTS00 - [email protected] [email protected] 0 0 5111 2006-2-7 10:28:41 GMT 0 - c=US;a= ;p=First Organizati;l=NTS00-060207102841Z-5 Will Public Folders disappear? EX:/O=FIRST ORGANIZATION/OU=FIRST ADMINISTRATIVE GROUP/CN=RECIPIENTS/CN=ILSE -
2006-2-7 10:28:41 GMT - - - NTS00 - [email protected] [email protected] 0 0 5111 2006-2-7 10:28:41 GMT 0 - - Will Public Folders
disappear? - -
2006-2-7 10:28:41 GMT - - - NTS00 - [email protected] [email protected] 0 0 5111 2006-2-7 10:28:41 GMT 0 - - Will Public Folders
disappear? - -
2006-2-7 10:28:41 GMT - - - NTS00 - [email protected] [email protected] 0 0 5111 2006-2-7 10:28:41 GMT 0 - - Will Public Folders
disappear? - -
2006-2-7 10:28:41 GMT - - - NTS00 - [email protected] [email protected] 0 0 5111 2006-2-7 10:28:41 GMT 0 - - Will Public Folders
disappear? [email protected] -
2006-2-7 10:28:41 GMT - - - NTS00 - [email protected] [email protected] 0 0 5111 2006-2-7 10:28:41 GMT 0 - - Will Public Folders
disappear? [email protected] -
Column DataDate 2006-2-7
Time 10:28:41 GMT
Client-IP -
Client-Hostname -
Partner-Name -
Server-Hostname NTS00
Server-IP -
Recipient-Address [email protected]
Event-ID 1027
MSGID [email protected]
Priority 0
Recipient-Report-Status 0
total-bytes 511
Number-Recipients 1
Origination Time 2006-2-7 10:28:41 GMT
Encryption 0
Service-Version -
Linked-MSGID c=US;a= ;p=First Organizati;l=NTS00-060207102841Z-5
Message-Subject Will Public Folders disappear?
Sender-Address EX:/O=FIRST ORGANIZATION/OU=FIRST ADMINISTRATIVE GROUP/CN=RECIPIENTS/CN=ILSE
Log ParserLog ParserLog Parser 2.2 can be used to reformat the tracking logs into a format digestible by SQL ServerLog Parser is available for download from the Microsoft Download Center:http://www.microsoft.com/download
Introducing Log ParserIntroducing Log Parser““The world is your database with Log Parser”The world is your database with Log Parser”
Log Parser allows users to treat log files and other information as SQL tables, the rows of which can be queried, processed, and formatted in different ways Born around 2000, as a utility to test the logging mechanisms of IISLatest release = version 2.2“Designed and engineered with the vision of helping users achieve their data-processing goals in a simple, fast, and powerful way.”
Building QueriesBuilding Queries
Data(Input Formats)
Output Records(Output Formats)
Log Parser Query
Building Blocks Log ParserBuilding Blocks Log ParserInput Format
Log Parser Query (dialect of SQL)Output Format
ADS ETW IIS NETMON TSV
BIN EVT IISODBC REG URLSCAN
COM FS IISW3C TEXTLINE W3C
CSV HTTPERR NCSA TEXTWORD XML
CHART IIS SYSLOG W3C
CSV NAT TPL XML
DATAGRID SQL TSV
Example: Retrieving some Example: Retrieving some fields from the Event Log fields from the Event Log
c:\LogParser –i:EVT –o:NAT “SELECT TimeGenerated, SourceName FROM System”
Or “SELECT TimeGenerated, SourceName INTOmytest.txt FROM System”
TimeGenerated SourceName------------------- -----------------------2005-11-10 12:26:07 Windows Update Agent2005-11-10 12:26:14 Windows Update Agent2005-11-10 15:00:23 Service Control Manager2005-11-10 15:00:23 Service Control Manager2005-11-10 15:00:44 Windows Update Agent2005-11-10 15:01:18 Windows Update Agent2005-11-10 15:01:30 NtServicePack2005-11-10 15:01:36 Windows Update Agent2005-11-10 15:01:50 Windows Update Agent2005-11-10 15:02:12 Windows Update AgentPress a key...
Back to Message Tracking Back to Message Tracking LogsLogs
Command:LogParser.exefile:f:\info\msgtracklog.sql?infile=f:\info\logs\20060207.log+outfile= f:\info\logs\20060207.bcp
-i:W3C -o:TSV
W3C input format parses log files in the W3C Extended Log File FormatTSV output format creates text file formatted according to the Tab-Seperated-Values convention
Log Parser Query SyntaxLog Parser Query Syntax
msgtracklog.sql:
SELECTTO_Timestamp(REPLACE_STR(STRCAT(STRCAT(date,' '), time),'
GMT',''),'yyyy-M-d h:m:s') as DateTime,[client-ip], [Client-hostname], [Partner-name], [Server-hostname], [server-
IP], [Recipient-Address],[Event-ID], [MSGID], [Priority], [Recipient-Report-Status], [total-bytes],
[Number-Recipients],TO_Timestamp(REPLACE_STR([Origination-time], ' GMT',''),'yyyy-M-d h:m:s')
as [Origination Time],Encryption, [service-Version], [Linked-MSGID], [Message-Subject], [Sender-
Address]INTO '%outfile%'FROM '%infile%'WHERE [Event-ID] IN (1027;1028)
Log Parser OutputLog Parser Output
DateTime client-ip Client-hostname Partner-Name Server-hostname server-IP Recipient-AddressEvent-IDMSGID Priority Recipient-Report-Status total-bytes Number-RecipientsOrigination TimeEncryption service-Version Linked-MSGID Message-SubjectSender-Address
2006-02-07 10:28:41 NTS00 [email protected] [email protected] 0 0 511 12006-02-07 10:28:41 0 c=US;a= ;p=First Organizati;l=NTS00-060207102841Z-5Will Public Folders disappear? EX:/O=FIRST ORGANIZATION/OU=FIRST ADMINISTRATIVE GROUP/CN=RECIPIENTS/CN=ILSE -
Column DataDateTime 2006-02-07 10:28:41
Client-IP -
Client-Hostname -
Partner-Name -
Server-Hostname NTS00
Server-IP -
Recipient-Address [email protected]
Event-ID 1027
MSGID [email protected]
Priority 0
Recipient-Report-Status 0
total-bytes 511
Number-Recipients 1
Origination Time 2006-02-07 10:28:41
Encryption 0
Service-Version -
Linked-MSGID c=US;a= ;p=First Organizati;l=NTS00-060207102841Z-5
Message-Subject Will Public Folders disappear?
Sender-Address EX:/O=FIRST ORGANIZATION/OU=FIRST ADMINISTRATIVE GROUP/CN=RECIPIENTS/CN=ILSE
Populate SQL with formed DATAPopulate SQL with formed DATA
Create a table to hold the data
Import information into the database
Several tools available, like:
SQL 2000: SQL Query Analyzer, osql utility, bcp utility
SQL 2005: SQL Management Studio, sqlcmd, bcp or osql utility
Create Table to Hold DataCreate Table to Hold DataUSE Analyzing_ExchangeCREATE TABLE MsgTrackingLogs (
[DateTime] datetime NULL ,[Client-IP] varchar (255) NULL ,[Client-Hostname] nvarchar (255) NULL ,[Partner-name] nvarchar (255) NULL ,[Server-hostname] nvarchar (255) NULL ,[Server-IP] varchar (255) NULL ,[Recipient-Address] varchar (512) NULL ,[Event-ID] int NULL ,[MSGID] nvarchar (1024) NULL ,[Priority] int NULL ,[Recipient-Report-Status] int NULL ,[Total-bytes] bigint NULL ,[Number-Recipients] int NULL ,[Origination Time] datetime NULL ,[Encryption] int NULL ,[Service-version] varchar (255) NULL ,[Linked-MSGID] varchar (255) NULL ,[Message-Subject] nvarchar (255) NULL ,[Sender-Address] varchar (255) NULL ,
)
Import Data Into DatabaseImport Data Into Database
Bcp Analyzing_Exchange.dbo.msgtrackinglogs
in f:\info\logs\20060207.bcp
–c
–t”\t”
–T
–F 2
Source 2: Active DirectorySource 2: Active DirectoryEach mailbox is an object in ADSome relevant properties:
legacyExchangeDNhomeMDBmDBUseDefaults (use default quota)mDBStorageQuota (issue warning)mDBOverQuotaLimit (prohibit send)mDBOverHardQuotaLimit (prohibit send/receive)Can also include fields like city, department, etc..
Use CSVDE to export data to CSV fileUse account with Exchange view only admin rights
Introducing CSVDEIntroducing CSVDE
csvde.exe installed on Windows 200X Server by defaultCan be run from Windows 2000 Pro or XP ProfessionalCan be used to import and export data from Active Directory by using files that store data in the comma-separated value (CSV) file format standardAlso supports batch operations that are based on CSV
CSVDE SyntaxCSVDE SyntaxCSVDE
-f file to export to-s servername-d LDAP search root-r LDAP search filter (default objectClass=*)-l list of attributes to export-u Unicode format (important for DBCS)
CSVDE ExampleCSVDE ExampleExtract a specified list of all Person objects in the
Matisse domain:
CSVDE –f f:\info\ad\directory.csv
-s NTSMATISSE
-d "dc=matisse,dc=edu"
-r "(&(objectCategory=Person)(homeMDB=*))"
-l DN,legacyExchangeDN,mail,homeMDB,mDBUseDefaults,mDBOverQuotaLimit,mDBStorageQuota,mDBOverHardQuotaLimit,department
-u
Create Table to Hold DataCreate Table to Hold DataCREATE TABLE [Active_Directory_Info] (
[DN] [varchar] (1000) NULL ,[legacyExchangeDN] [varchar] (512) NULL,[mail] [varchar] (512) NULL ,[homeMDB] [varchar] (1000) NULL ,[mDBUseDefaults] [varchar] (10) NULL ,[mDBOverQuotaLimit] [int] NULL ,[mDBStorageQuota] [int] NULL ,[mDBOverHardQuotaLimit] [int] NULL ,[Department] [varchar] (256) NULL ,
) ON [PRIMARY]
Import Data Into DatabaseImport Data Into DatabaseLogParser
"SELECT DN,legacyExchangeDN,mail,homeMDB,mDBUseDefaults,mDBOverQuotaLimit,mDBStorageQuota,mDBOverHardQuotaLimit,department
into dbo.Active_Directory_Info
FROM f:\info\ad\directory.csv"
-i:csv
-o:SQL -server:servername -database:Analyzing_Exchange -driver:"SQL Server"
Source 3: WMI Source 3: WMI Windows Management InstrumentationManagement technology allowing scripts to monitor and control managed resources throughout the networkResources include hard drives, file systems, operating system settings, processes, services, shares, registry settings, networking components, event logs, users, and groupsBuilt into clients with Windows 2000 or above, and can be installed on any other 32-bit Windows clientWMI is easy to consume via script
Exchange_Mailbox WMI ClassExchange_Mailbox WMI ClassNew class for Exchange 2003
Returns properties of a mailbox
Interesting fields:MailboxDisplayName
LegacyDN (legacyExchangeDN)
ServerName (Exchange server name)
Size (size of mailbox in kb)
TotalItems (total # messages in the mailbox)
DeletedMessageSizeExtended (Size in bytes of deleted messages being retained per deleted items retention policy)
VBScript to Access WMI DataVBScript to Access WMI DatastrWinMgmts =
"winmgmts:{impersonationLevel=impersonate}!//ServerName/root/MicrosoftExchangeV2"
Set objWMIExchange = GetObject(strWinMgmts)Set listExchange_Mailboxes = objWMIExchange.InstancesOf("Exchange_Mailbox")For each objExchange_Mailbox in listExchange_Mailboxes Wscript.echo objExchange_Mailbox.MailboxDisplayName & vbTab _
& objExchange_Mailbox.LegacyDN & vbTab _& objExchange_Mailbox.ServerName & vbTab _& objExchange_Mailbox.Size & vbTab _& objExchange_Mailbox.TotalItems & vbTab _
& objExchange_Mailbox.DeletedMessageSizeExtended & vbTab _ & objExchange_Mailbox.LastLogonTime & vbTab _ & objExchange_Mailbox.LastLogOffTime & vbTab _
& objExchange_Mailbox.LastLoggedOnUserAccountNext
Execute as: cscript //nologo mailboxes.vbs > Mailboxes.txt
Create Table to Hold DataCreate Table to Hold Data
CREATE TABLE [MailboxSizeData] ([displayName] [varchar] (128) NULL ,[legacyExchangeDN] [varchar] (512) NULL ,[ServerName] [varchar] (50) NULL ,[Size] [int] NULL ,[TotalItems] [int] NULL ,[DeletedMessageSizeExtended] [int] NULL ,[LastLogonTime] [varchar] (50) NULL ,[LastLogoffTime] [varchar] (50) NULL ,[LastLoggedOnUserAccount] [varchar] (50) NULL
) ON [PRIMARY]
Import Data Into DatabaseImport Data Into DatabaseMicrosoft SQL Server Management Studio “BULK INSERT [MailboxSizeData] FROM ‘f:\
info\wmi\Mailboxes.txt'“
AgendaAgenda
Introduction
Process flow
Sources of data
Pulling it all together: Joins
Report creation
Automation
Pulling it all togetherPulling it all together
SQL joins let us relate data in one table with data in another tablePowerful feature for rich reportsUse common columns to relate data
Table JoinsTable Joins
MsgTrackingLogs, MailboxSizeData and Active_Directory_Info can all be joinedActive_Directory_Info and MailboxSizeData join on [legacyExchangeDN]Then join Active_Directory_Info to MsgTrackingLogs on [Recipient-Address]
JoinJoin
AgendaAgenda
Introduction
Process flow
Sources of data
Pulling it all together: Joins
Report creation
Automation
Report Creation – Getting Report Creation – Getting StartedStarted
Install Reporting Services Install Reporting Services Client ToolsCreate new “Business Intelligence” projectSelect template: “Report Server Project Wizard”
Six Easy StepsSix Easy Steps
1. Define data source
2. Design query
3. Choose type of report
4. Specify basic layout of report
5. Format the report
6. Deploy the report
DeployDeployCheck project propertiesCheck TargetServerUrRL =http://myserver.mydomain.com/ReportServer
Deploy!When complete, browse with IE:
http://servername/Reports
AgendaAgenda
Introduction
Process flow
Sources of data
Pulling it all together: Joins
Report creation
Automation
AutomationAutomation
Automate data gathering and import with:
SQL 2000: Data Transformation Services (DTS)
SQL 2005: SQL System Integration Services
(SSIS)
SummarySummary
Reports are vital to the health of your messaging infrastructure
Basic reports are fairly easy to develop
Three key data sources: AD, Tracking logs, WMI
This session gave you a very limited view of all the power you have when you use the available tools to create custom reports!
Thank you for your Thank you for your attention!attention!
Ilse Van CriekingeExchange Trainer & Consultant
(Azlan Training)
ResourcesResourcesLogParser 2.2
Microsoft Log Parser Toolkit, Gabriele Giuseppini&Mark Burnett, Syngresshttp://www.logparser.com/http://www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=enhttp://www.microsoft.com/technet/scriptcenter/tools/logparser/default.mspx
Message tracking log field descriptionshttp://support.microsoft.com/default.aspx?scid=kb;en-us;246965
Message tracking log event id definitionshttp://support.microsoft.com/default.aspx?scid=kb;en-us;821905
SQL Server 2000/2005http://www.microsoft.com/sql
Visual Studio 2005http://msdn.microsoft.com/vstudio/
WMI Exchange_Mailbox classhttp://msdn.microsoft.com/library/default.asp?url=/library/en-us/e2k3/e2k3/_wmiref_cl_Exchange_Mailbox.asp
The Connected GenerationThe Connected Generation
7 & 8 March 20067 & 8 March 2006
ICC GentICC Gent
