Oracle Trace File Analyzer (TFA) · • Lightweight tool that runs within your environment without...
Transcript of Oracle Trace File Analyzer (TFA) · • Lightweight tool that runs within your environment without...
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle Trace File Analyzer(TFA)Diagnostics for the Cloud
12.2.1.3.0
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 3
Easy to use real-timehealth monitoring, fault detection & diagnosis via a single interface
Secure consolidation of distributed diagnostic collections
Continuous availability so you always get what’s needed for resolution
Machine learning driven, autonomous degradation detection, reduces your overheads
Why TFA?
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 4
Autonomous Usage
Oracle Grid Infrastructure& Databases
Oracle Support
TFA
1
TFA detects a fault
2Diagnosticsare collected
3Distributed diagnostics are consolidated and packaged
4Notification of fault is sent
5 Diagnostic collection is uploaded to Oracle Support for root cause analysis & resolution
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 5
On-Demand Usage
Oracle Grid Infrastructure& Databases
Oracle Support
TFA
5
Upload diagnostic collection to Oracle
Support
2Real-time
statussummary
3Diagnose
with DB tools
4Perform
diagnostic collection
1 Request desired action on-demand
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Specify all command options at the command line
1. Set and change context
2. Run commands from within the shell
6
1. Select menu navigation options then choose the command you want to run
Command Interfaces
Command line Shell Menu
tfactl <command>
tfactl
tfaclt > database MyDB
MyDB tfactl > oratop
tfactl menu
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
mask_strings.xml • If you want to mask sensitive data– Hostname
– IP address
– Etc
• Create mask_strings.xml– In the directory tfa_home/resources
– Define all data replacements you want
7
Data Redaction
<mask_strings>
<mask_string>
<original>WidgetNode1</original>
<replacement>Node1</replacement>
</mask_string>
<mask_string>
<original>192.168.5.1</original>
<replacement>Node1-IP</replacement>
</mask_string>
<mask_string>
<original>WidgetNode2</original>
<replacement>Node2</replacement>
</mask_string>
<mask_string>
<original>192.168.5.2</original>
<replacement>Node2-IP</replacement>
</mask_string>
</mask_strings>
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Enable or disable user access
Configure User AccessUsers & groups authorized to access tfactl by default
tfactl access enable
tfactl access disable
tfactl access add –user <user> [-local]
tfactl access remove –user <user> [-local]
tfactl access removeall [-local]
• Add users
• Remove users or everyone
All commands apply cluster wide unless -local used to restrict to local node
• Reset to default access
tfactl access reset
• List user access
tfactl access lsusers
8
TFA
Grid Home Owner DB Home Owners Everyone Else
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Set Email Notification Addresses
9
tfactl set [email protected]
tfactl set notificationAddress=oracle:[email protected]
• TFA can send email notification when faults are detected
• To set notification email for any problem detected:
• To set notification email for specific ORACLE_HOMEs include the OS owner:
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 10
On-Demand Usage
Oracle Grid Infrastructure& Databases
Oracle Support
TFA
5
Upload diagnostic collection to Oracle
Support
2Real-time
statussummary
3Diagnose
with DB tools
4Perform
diagnostic collection
1 Request desired action on-demand
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 11
Real-time Status Summary
tfactl summary
Choose an option to drill down
High-level summary of all Database components
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 12
Real-time Status Summary – Drill Down
Drill downs show real-time analytics & details of any problems found
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 13
On-Demand Usage
Oracle Grid Infrastructure& Databases
Oracle Support
TFA
5
Upload diagnostic collection to Oracle
Support
2Real-time
statussummary
3Diagnose
with DB tools
4Perform
diagnostic collection
1 Request desired action on-demand
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Analyze all important recent log entries: • Search recent log entries:
14
Investigate Logs & Look for Errors
tfactl analyze –last 1d tfactl analyze -search “ora-006" -last 8h
Searching for “ora-00600”
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Perform Analysis Using the Included Tools
Not all tools are included in Grid or Database install.Download from 1513912.1 to get full collection of tools
Tool Description
orachk or exachk
Provides health checks for the Oracle stack. Oracle Trace File Analyzer will install either • Oracle EXAchk for Engineered Systems, see document 1070954.1 for
more detailsor • Oracle ORAchk for all non-Engineered Systems, see document
1268927.2 for more details
oswatcher Collects and archives OS metrics. These are useful for instance or node evictions & performance Issues. See document 301137.1 for more details
procwatcher Automates & captures database performance diagnostics and session level hang information. See document 459694.1 for more details
oratop Provides near real-time database monitoring. See document 1500864.1for more details.
alertsummary Provides summary of events for one or more database or ASM alert files from all nodes
ls Lists all files TFA knows about for a given file name pattern across all nodes
pstack Generate process stack for specified processes across all nodes
Tool Description
grep Search alert or trace files with a given database and file name pattern, for a search string.
summary Provides high level summary of the configuration
vi Opens alert or trace files for viewing a given database and file name pattern in the vi editor
tail Runs a tail on an alert or trace files for a given database and file name pattern
param Shows all database and OS parameters that match a specified pattern
dbglevel Sets and unsets multiple CRS trace levels with one command
history Shows the shell history for the tfactl shell
changes Reports changes in the system setup over a given time period. This includes database parameters, OS parameters and patches applied
calog Reports major events from the Cluster Event log
events Reports warnings and errors seen in the logs
managelogs Shows disk space usage and purges ADR log and trace files
ps Finds processes
triage Summarize oswatcher/exawatcher data
15
Verify which tools you have installed: tfactl toolstatus
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 16
Oracle Stack Health Checks
• Oracle EXAchk for Engineered Systems
• Oracle ORAchk for everything else• Proactively scans for the most impactful problems across the layers of your stack
• Simplifies and streamlines how to investigate and analyze which known issues present a risk to you
• Lightweight tool that runs within your environment without requiring config data to be sent to Oracle
• High level reports show your system health risks with the ability to drill down into specific problems and understand their resolutions
• Can be configured to send email notifications when it detects problems
• Collection Manager, a companion Application Express web app, provides a single dashboard view of collections across your entire enterprise
ORAchk & EXAchk (Support Tools Bundle)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 17
OS Watcher (Support Tools Bundle)
Collect & Archive OS Metrics• Executes standard UNIX utilities (e.g. vmstat, iostat, ps,
etc) on regular intervals
• Built in Analyzer functionality to summarize, graph and report upon collected metrics
• Output is Required for node reboot and performance issues
• Simple to install, extremely lightweight
• Runs on ALL platforms (Except Windows)
• MOS Note: 301137.1 – OS Watcher Users Guide
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 18
Procwatcher (Support Tools Bundle)
Monitor & Examine Database Processes
• Single instance & RAC
• Generates session wait, lock and latch reports as well as call stacks from any problem process(s)
• Ability to collect stack traces of specific processes using Oracle Tools and OS Debuggers
• Typically reduces SR resolution for performance related issues
• Runs on ALL major UNIX Platforms
• MOS Note: 459694.1 – Procwatcher Install Guide
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 19
oratop (Support Tools Bundle)
Near Real-Time Database Monitoring
• Single instance & RAC
• Monitoring current database activities
• Database performance
• Identifying contentions and bottleneck
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Analyze
• Each tool can be run using tfactl in shell mode
• Start tfactl shell with
• Run a tool with the tool name
1. Where necessary set context with database <dbname>
2. Then run tool
3. Clear context with database
20
tfactl
tfactl > database MyDB
MyDB tfactl > database
tfactl > orachk
MyDB tfactl > oratop
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Automatic Database Log Purge
• TFA can automatically purge database logs–OFF by default
– Except on a Domain Service Cluster (DSC), which it is ON by default
• Turn auto purging on or off:
• Will remove logs older than 30 days– configurable with:
• Purging runs every 60 minutes– configurable with:
21
tfactl set manageLogsAutoPurge=<ON|OFF>
tfactl set manageLogsAutoPurgePolicyAge=<n><d|h>
tfactl set manageLogsAutoPurgeInterval=<minutes>
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Manual Database Log Purge
• TFA can manage ADR log and trace files
– Show disk space usage of individual diagnostic destinations
– Purge these file types based on diagnostic location and or age:
• "ALERT“, "INCIDENT“, "TRACE“, "CDUMP“, "HM“, "UTSCDMP“, "LOG“
tfactl managelogs <options>
Runs as the ADR home owner. So will only be able to purge files this owner has permission to delete
Option Description
–show usage Shows disk space usage per diagnostic directory for both GI and database logs
-show variation –older <n><m|h|d> Use to determine per directory disk space growth.
Shows the disk usage variation for the specified period per directory.
-purge –older <n><m|h|d> Remove all ADR files under the GI_BASE directory, which are older than the time specified
–gi Restrict command to only diagnostic files under the GI_BASE
–database [all | dbname] Restrict command to only diagnostic files under the database directory. Defaults to all,
alternatively specify a database name
-dryrun Use with –purge to estimate how many files will be affected and how much disk space will be
freed by a potential purge command.
May take a while for a large number of files
22
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 23
Manual Database Log Purge
tfactl managelogs –show usage tfactl managelogs –show variation –older <n><m|h|d>
Use -gi to only show grid infrastructure
Use –database to only show database
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 24
Manual Database Log Purgetfactl managelogs –purge –older n<m|h|d> -dryrun tfactl managelogs –purge –older n<m|h|d>
Use –dryrunfor a “what if”
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Disk Usage Snapshots
• TFA will track disk usage and record snapshots to:– tfa/repository/suptools/<node>/managelogs/usage_snapshot/
• Snapshot happens every 60 minutes, configurable with:
• Disk usage monitoring is ON by default, configurable with:
25
tfactl set diskUsageMonInterval=<minutes>
tfactl set diskUsageMon=<ON|OFF>
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Change Trace Levels
• Sometimes Oracle Support may require certain specific trace levels to be enabled when reproducing a problem
• TFA makes it easy to enable and disable the correct trace levels with a single command
• All the required trace level settings are organised into problem specific profiles
• The relevant profile trace logging can be enabled with:
26
tfactl dbglevel –set <profile>
• Profiles only available for Clusterware today, Database profiles coming soon..
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 27
On-Demand Usage
Oracle Grid Infrastructure& Databases
Oracle Support
TFA
5
Upload diagnostic collection to Oracle
Support
2Real-time
statussummary
3Diagnose
with DB tools
4Perform
diagnostic collection
1 Request desired action on-demand
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• For certain types of problems Oracle Support will ask you to run a Service Request Data Collection (SRDC)
• Previously this would have involved:
• Reading many different support documents
• Collecting output from many different tasks
• Gathering lots of different diagnostics
• Packaging & uploading
• Now just run:
One Command SRDCs
tfactl diagcollect -srdc <srdc_type> -sr <SR#>
28
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 29
Collection Uploaded to My Oracle Support
At the end of the collection the file is automatically uploaded to the SR
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
With Wallet: Without wallet:
30
Upload any File
tfactl upload -wallet -sr <sr_number> <file1 file2> tfactl upload -user <user_id> -sr <sr_number> <file1 file2>
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 31
Faster & Easier SR Data Collectiontfactl diagcollect –srdc <srdc_type> -sr <SR#>
Type of Problem SRDC
ORA Errors
• ORA-00020
• ORA-00060
• ORA-00600
• ORA-00700
• ORA-01555
• ORA-01628
• ORA-04030
• ORA-04031
• ORA-07445
• ORA-27300
• ORA-27301
• ORA-27302
• ORA-30036
Other internal database errors • internalerror
Database performance • dbperf
Database patching• dbpatchinstall
• dbpatchconflict
Database resource • dbunixresources
XDB installation or invalid object • dbxdb
Database install / upgrade
• dbinstall
• dbupgrade
• dbpreupgrade
Type of Problem SRDCDatabase storage • asm
Excessive SYSAUX Space used by the Automatic
Workload Repository (AWR)
• dbawrspace
Database startup / shutdown • dbshutdown
• dbstartup
Data Guard • dbdataguard
Enterprise Manager tablespace usage metric • emtbsmetrics
Enterprise Manager general metrics page or
threshold problems - Run all three SRDCs
• emdebugon
• emdebugoff
• emmetricalert
Enterprise Manager target discovery / add
• emcliadd
• emclusdisc
• emdbsys
• emgendisc
• emprocdisc
Enterprise Manager OMS restart • emrestartoms
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
ORA-04031:
1. IPS Package
2. Patch Listing
3. AWR report
4. Memory information
5. RDA
Database Performance
1. ADDM report
2. AWR for good and problem period
3. AWR Compare Period report
4. ASH report for good and problem period
5. OS Watcher
6. IPS Package (if errors during problem period)
7. ORAchk (performance related checks)
One Command SRDCs – Examples of What’s Collected
tfactl diagcollect –srdc ORA-04031 tfactl diagcollect –srdc dbperf
32
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Manual Data Gathering vs One Command SRDC
Manual Data Gathering
1. Generate ADDM reviewing Document 1680075.1
2. Identify “good” and “problem” periods and gather AWR reviewing Document 1903158.1
3. Generate AWR compare report (awrddrpt.sql) using “good” and “problem” periods
4. Generate ASH report for “good” and “problem” periods reviewing Document 1903145.1
5. Collect OSWatcher data reviewing Document 301137.1
6. Check alert.log if there are any errors during the “problem” period
7. Find any trace files generated during the “problem” period
8. Collate and upload all the above files/outputs to SR
TFA SRDC
1. Run
2. Upload resulting zip file to SR
tfactl diagcollect –srdc dbperf
33
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
1. Enter default for event date/time and database name
2. Scans system to identify recent 10 events in the system (ORA600 example shown)
3. Once the relevant event is chosen, proceeds with diagnostic collection
34
One Command SRDC
tfactl diagcollect –srdc <srdc_type>
4. All required files are identified
5. Trimmed where applicable
6. Package in a zip ready to provide to support
Interactive Mode
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
One Command SRDC
35
tfactl diagcollect –srdc <srdc_type> -database <db> -for <time>
1. Parameters(date/time, DB name) are provided in the command
2. Does not prompt for any more information
3. All required files are identified
4. Trimmed where applicable
5. Package in a zip ready to provide to support
Silent Mode
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Run a default diagnostic collection if there is not yet an SRDC about your problem:
• Will trim & collect all important log files updated in the past 12 hours:
• Collections stored in the repository directory
• Change diagcollect timeframe with
–last <n>h|d
36
Default Collection
tfactl diagcollect
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 37
Time
Custom Collection Options
Option Description
-last <n><h|d> Collect since the previous n hours or
days
-from “<yyyy-mm-dd>” Collect from the date and optionally
time specified.
Valid date / time formats:
"Mon/dd/yyyy hh:mm:ss"
or "yyyy-mm-dd hh:mm:ss"
or "yyyy-mm-ddThh:mm:ss"
or "yyyy-mm-dd "
Option Description
–from "yyyy-mm-dd"
-to "yyyy-mm-dd"
Collect to the date and optionally time
specified.
Valid date / time formats:
"Mon/dd/yyyy hh:mm:ss"
or "yyyy-mm-dd hh:mm:ss"
or "yyyy-mm-ddThh:mm:ss"
or "yyyy-mm-dd"
-for “yyyy-mm-dd” Collect for the specified date.
Valid date formats:
"Mon/dd/yyyy"
or "yyyy-mm-dd"
tfactl diagcollect <option>
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 38
Nodes, Databases & Components
Custom Collection Options
Option Description
-node Limit collection to comma separated list of nodes
-database Limit collection to comma separated list of databases
-asm Collect ASM logs
-crsclient Collect Client Logs that are under GIBASE/diag/clients
-dbclient Collect Client Logs that are under DB ORABASE/diag/clients
-dbwlm Collect DBWLM logs
-tns Collect TNS logs
-rhp Collect RHP logs
-procinfo Collect Gathers stack and fd from /proc for all processes
-afd Collect AFD logs
-crs Collect CRS logs
-wls Collect WLS logs
Option Description
-emagent Collect EMAGENT logs
-oms Collect OMS logs
-ocm Collect OCM logs
-emplugins Collect EMPLUGINS logs
-em Collect EM logs
-acfs Collect ACFS logs and Data
-install Collect Oracle Installation related files
-cfgtools Collect CFGTOOLS logs
-os Collect OS files such as /var/log/messages
-ashhtml Collect Generate ASH HTML Report
-ashtext Collect Generate ASH TEXT Report
-awrhtml Collect AWRHTML logs (requires –database option as well)
tfactl diagcollect <option>
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 39
Misc
Custom Collection Options
Option Description
-collectdir Collects other files in specified comma separated directory list. Only files updated in the relevant time
range (12 hours by default) get collected.
-tag Store the collection in a use a custom directory name within the repository, using the tag value specified
-z Append the value specified to the collection zip name
-nocopy Prevent the copying of zip files back to the initiating node
-notrim Prevent the trimming of files
-silent Start collection as a background process so output is not echoed to the console
-nocores Do not collect core files
tfactl diagcollect <option>
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• TFA can invoke IPS to query and collect these packages
Simple IPS (Incident Packaging Service) interface
• Database incidents are stored in the ADR (Automatic Diagnostic Repository)
• IPS packages details about these incidents for later diagnosis
40
Command Description
tfactl ips Invoke IPS
tfactl ips show incidents Show all IPS incidents
tfactl ips show problems Show all IPS problems
tfactl ips show package Show all IPS packages
tfactl diagcollect -ips –help Show all available diagcollect IPS options
tfactl diagcollect –ips Perform an IPS collection following prompts
tfactl diagcollect -ips -adrbasepath
<adr_base> -adrhomepath <adr_home>
Perform an IPS collection in silent mode.
tfactl diagcollect -ips -incident <incident id> Collect ADR details about a specific incident id
tfactl diagcollect -ips -problem <problem id> collect ADR details about a specific problem id
tfactl ips <options>
tfactl diagcollect –ips <options>
or
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Display details of incidents recorded in ADR
41
Show Incidents
tfactl ips show incidents
• Incident ID
• Problem Key
• Create Time
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Display details of problems recorded in ADR
42
Show Problems
tfactl ips show problems
• Problem ID
• Problem Key
• Last Incident
• Last Incident Time
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
• Display details of packages recorded in ADR
43
Show Package
tfactl ips show package
List of packages &
details of each package
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 44
Collect Packages
tfactl diagcollect -ips
Select ADR home
Generate package
Rest of collection
Resulting collection zips
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Maintenance
• Option 1– Applying standard PSUs will automatically
update TFA
– PSUs do not contain Support Tools Bundle updates
• Option 2– To update with latest TFA & Support Tools
Bundle 1513912.1
45
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
46