2012 Feb EBS Concurrent Processing Troubleshooting


My Oracle Support – Advisor Webcast Program

• EBS Concurrent Processing Troubleshooting

Techniques Wednesday, 15 February, 2012 06:00 PM CET (GMT +2) Teleconference Access: North America: 1866 966 9439 International (UK): +44 (0) 1452 562 665 Conference ID : 39485824

Upcoming Webcasts in EBS Technology area :

o March 2012 - EBS Patching Overview & Troubleshooting

Do you have any requests for future EBS - Technology

Webcast Events? Please email your suggestions to

[email protected],subject: Topics of Interest.

mailto:[email protected]


AGENDA

• Presentation and Demo

approximately 60 - 75 minutes

• Q&A Session

maximum 15 minutes

Web attendees can ask questions via Q&A panel

Phone attendees can ask questions via Q&A panel or phone


Q&A panel

type your question here 1

Send

your question

3

Who to ask? 2


your question pops-up here


ATTENTION – AUDIO INFORMATION

• Voice streaming/Audio broadcast is available. For full audio access,

• please join the telephone conference.

Teleconference Connect details:

1. Conference ID: 39485824

2. International dial in: +44 (0) 1452 562 665

3. List with national toll free numbers is available in Note 1148600.1

Note:

You can view this info anytime using WebEx menu from your WebEx-Session :

Select ‘Communicate’ --> ‘Join Teleconference’.


The following is intended to outline our general product direction.

It is intended for information purposes only, and may not be

incorporated into any contract. It is not a commitment to deliver any

material, code, or functionality, and should not be relied upon in making

purchasing decisions. The development, release, and timing of any

features or functionality described for Oracle’s products remains at the

sole discretion of Oracle.

Safe Harbor Statement


EBS Concurrent Processing Troubleshooting Techniques

Presenters: Kandasamy Nainamalai, Kalaivani Murugesan and Rakesh Chithuluri

Q&A Panel Experts : Amira Abdelkader , Vishwanatha Rao,Thiagarajan, Martin Fritz


Program Agenda • Concurrent Manager - Troubleshooting. 1. Manager Not Running (inactive NoManager)

2. How to run cmclean(Pending Requests)

3. ICM Running , Other Managers are down

4. Only OPP & Workflow services Down

5. DBMS_LOCK Error - ICM Log

6. After CLONE Managers Down/FNDSM Errors/CONC-SM TNS FAIL

7. Managers down - Status "System Hold, Fix Manager"

8. Managers down - "Target node/queue unavailable“

9. How to Enable diag=y

• Concurrent Request – Troubleshooting 1. ORA-20100 Error Troubleshooting

2. Rep-3000 Error troubleshooting

3. Unable to View output/log files

4. How to Collect a Trace and Tkprof

• XML Publisher Reports - Troubleshooting 1. XML Publisher Introduction

2. How to Obtain OPP Log

3. How to enable & collect XDO Log

4. BI Publisher - Troubleshooting


Concurrent Manager - Troubleshooting.

1. Manager Not Running (inactive NoManager)

2. How to run cmclean(Pending Requests)


Concurrent Manager – Troubleshooting Concurrent Request Stays “Inactive NoManager”


• System Administrator Responsibility -> Concurrent -> Manager ->

• Administer Form -> Verify the Concurrent Managers Status

• Login into Backend CP Node

Verify ps –ef | FNDLIBR process

• Start Concurrent Manager (adcmctl.sh)

• Ensure All Managers are running now.

Manager Not Running (Inactive NoManager)


• Ensure Concurrent Managers are down.

• Collect cmclean.sql script from below document.

Note.134007.1 Ext/Pub CMCLEAN.SQL - Non Destructive Script to Clean Concurrent Manager Tables

• Execute the cmclean.sql script.

• Start the Concurrent Manager (adcmctl.sh)

How to Run CMCLEAN


Concurrent Manager Troubleshooting

Demonstration – part1


Concurrent Manager - Troubleshooting.

1. ICM Running , Other Managers are down

2. Only OPP & Workflow services Down

3. DBMS_LOCK Error - ICM Log

4. After CLONE Managers Down/

FNDSM Errors/CONC-SM TNS FAIL

5. Managers down - Status "System Hold, Fix Manager"

6. Managers down - "Target node/queue unavailable“

7. How to Enable diag=y


ICM Running , Other Managers are down


Solution :

• 1. Check listener Status (adalnctl.sh status)

• 2. Stop Concurrent Manager (adcmctl.sh stop)

• 3. Start Listener (adalnctl.sh start)

• 4. Start Concurrent Manager (adcmctl.sh start)


Concurrent Manager Troubleshooting

Demonstration – part2


Only OPP & Workflow services Down


Solution :

• Ensure Concurrent:GSM Enabled profile is set to ‘Y’

• Verify Service Manager status in Administer Form.

• Verify Service Manager Definition.

• Ensure FNDSM Entries available in FND_CONCURRENT_QUEUES Table

• FNDSM entry should be correct in Tnsnames.ora file and tnsping

FNDSM_hostname should work fine.

• Then Bounce the Services.


DBMS_LOCK Error – Internal Concurrent Manager Log


Solution :

• Ensure Concurrent Managers are down.

Ensure no FNDLIBR process running

ps –ef|grep FNDLIBR

• Run Cmclea.sql ( commit is mandatory)

• Then Start the services – adstrtal.sh apps/apps

• Now you can see Managers Up and running.


After CLONE Managers Down

• After CLONE Concurrent Manager not Starting

ICM logfile shows the following errors:

Error#1:

CONC-SM TNS FAIL

Call to StopProcess failed for FNDCPGSC

CONC-SM TNS FAIL

Call to PingProcess failed for FNDOPP

Error#2:

Could not contact Service Manager FNDSM_DHSRVMDS518_SDBA. The TNS alias could not be located, the listener process

on DHSRVMDS518 could not be contacted, or the listener failed to spawn the Service Manager process.

Error#3:

Cause: cleanup_node failed due to ORA-01427: single-row subquery returns more than one row

ORA-06512: at "APPS.FND_CP_FNDSM", line 29

ORA-06512: at line 1.


Solution:

• Ensure Database is running

• Ensure Managers are down.

• Connect SQLPLUS as APPS user and run the following :

EXEC FND_CONC_CLONE.SETUP_CLEAN;

COMMIT;

EXIT;

• Run Autoconfig on DB node and then Apps Node

• Run $FND_TOP/patch/115/sql/afdcm037.sql script

• Start Services (adstrtal.sh)


References:

• Concurrent Processing - After Cloning all the Concurrent Managers do not start for the cloned

Instance (Doc ID 555081.1)

• ICM log file shows 'CONC-SM TNS FAIL', 'Call to PingProcess failed', and/or 'Call to StopProcess

failed' for FNDCPGSC/FNDOPP (Doc ID 1312632.1)


Managers down - Status "System Hold, Fix Manager"


Solution:

• Ensure Concurrent :GSM Enabled profile is set to ‘Y’

• Run $FND_TOP/patch/115/sql/afdcm037.sql

• Go to $FND_TOP/bin

adrelink.sh force=y “fnd FNDLIBR”

adrelink.sh force=y “fnd FNDSM”

adrelink.sh force=y “fnd FNDFS”

adrelink.sh force=y “fnd FNDCRM”

• Run cmclean.sql

• Start Application Service (adstrtal.sh)


Managers down – “Target node/queue unavailable”


Solution:

Ensure Database is Running and Middle Tier Services are down.

o Connect SQLPLUS as APPS user and run the following :

EXEC FND_CONC_CLONE.SETUP_CLEAN;

COMMIT;

EXIT;

o Run AutoConfig on all tiers, firstly on the DB tier and then the APPS tiers and webtier to repopulate the required system

tables

o Run the CMCLEAN.SQL script from the referenced note below (don't forget to commit).

Note.134007.1 – ‘CMCLEAN.SQL - Non Destructive Script to Clean Concurrent Manager Tables‘

o Start the middle tier services including your concurrent manager.

o Retest the issue.


How to Enable diag=y

• Stop Concurrent Manager

• Take backup of adcmctl.sh

• Edit adcmctl.sh to append diag=y

$FND_TOP/bin/startmgr sysmgr="$unpw@$DB_NAME" $@ mgrname=$MGR_NAME diag=y

else

$FND_TOP/bin/startmgr sysmgr="$unpw@$DB_NAME" mgrname=$MGR_NAME diag=y

fi;

• Start Concurrent Manager


Concurrent Request – Troubleshooting

1. ORA-20100 Error Troubleshooting

2. Rep-3000 Error troubleshooting

3. Unable to View output/log files

4. How to Collect a Trace and Tkprof


Concurrent Request Troubleshooting

Demonstration – Part1


ORA-20100 Error Troubleshooting


Solution:

• Ensure Tmp Directory is valid. ($APPLPTMP)

• Tmp Directory should have enough space.

• Applmgr user does not have valid permission (Read/Write/Execute)

• Different instances use same Tmp Directory

• UTL_FILE_DIR Database Parameter does not have APPLPTMP Entry


Rep-3000 Error troubleshooting

Any Bitmap Reports Failing with following Error:

APPLLCSP Environment Variable set to :

XML_REPORTS_XENVIRONMENT is :

/u01/app/protoora/8.0.6/guicommon6/tk60/admin/Tk2Motif_UTF8.rgb

XENVIRONMENT is set to: /u01/app/protoora/8.0.6/guicommon6/tk60/admin/Tk2Motif_UTF8.rgb

Current NLS_LANG and NLS_NUMERIC_CHARACTERS Environment Variables are :

American_America.UTF8

' '

REP-3000: Internal error starting Oracle Toolkit.

REP-3000: Internal error starting Oracle Toolkit.


Solution:

• Ensure an Xserver configured for CP server

• Ensure DISPLAY is set to valid Xserver

• Set DISPLAY in Xserver as root user and confirm Xclock is working

• Run xhost +

• From Backend CP Node Set DISPLAY as applmgr User

• Run Xclock from CP Node to verify connection between CP Node and Xserver

• Go to Front end and Verify DISPLAY Value

• If DISPLAY is incorrect in Front end then,

• Go to Oracle Application Manager and Change the s_display.

• Then Run Autoconfig

• Bounce Concurrent Manager.


Unable to View output/log files

Ensure RRA: Enabled profile is Set to ‘Yes’


Unable to View output/log files

• Ensure Listener is Running fine.

adalnctl.sh status


How to Collect a Trace and Tkprof

• Program Level Trace (Not Recommended)

Go to Concurrent Program -> Define form -> Query for Concurrent Program Enable Trace

• Request Level Trace (Recommended)

Ensure Concurrent: Allow Debugging profile is Set to ‘Yes’

On Submission Request Form -> Click on Debug Options Button

Select the Required Trace Level -> Click Ok.


Get Trace File Name Run the following SQL to find out the Raw trace name and location for the concurrent program. The SQL prompts the user for the request id

prompt

accept request prompt 'Please enter the concurrent request id for the appropriate concurrent program:'

prompt

column traceid format a8

column tracename format a80

column user_concurrent_program_name format a40

column execname format a15

column enable_trace format a12

set lines 80

set pages 22

set head off

SELECT 'Request id: '||request_id ,

'Trace id: '||oracle_Process_id,

'Trace Flag: '||req.enable_trace,

'Trace Name:

'||dest.value||'/'||lower(dbnm.value)||'_ora_'||oracle_process_id||'.trc',

'Prog. Name: '||prog.user_concurrent_program_name,

'File Name: '||execname.execution_file_name|| execname.subroutine_name ,

'Status : '||decode(phase_code,'R','Running')

||'-'||decode(status_code,'R','Normal'),

'SID Serial: '||ses.sid||','|| ses.serial#,

'Module : '||ses.module

from fnd_concurrent_requests req, v$session ses, v$process proc,

v$parameter dest, v$parameter dbnm, fnd_concurrent_programs_vl prog,

fnd_executables execname

where req.request_id = &request

and req.oracle_process_id=proc.spid(+)

and proc.addr = ses.paddr(+)

and dest.name='user_dump_dest'

and dbnm.name='db_name'

and req.concurrent_program_id = prog.concurrent_program_id

and req.program_application_id = prog.application_id

--- and prog.application_id = execname.application_id

and prog.executable_application_id = execname.application_id

and prog.executable_id=execname.executable_id;


Get TkProf File

Syntax :

• $ tkprof <Trace Filename> <Tkprof Filename> explain=apps/password sort=‘(prsela,exeela,fchela)’


XML Publisher Reports

Introduction to XML Publisher Reports

OPP log file

XDO log file

Troubleshooting


XML Publisher - Troubleshooting

Demonstration


OPP Log File

Possible ways to obtain the OPP log file:

1. Directly from the file system based after identifying the corresponding OPP log file name using the following SQL

statement:

SELECT fcpp.concurrent_request_id req_id, fcp.node_name, fcp.logfile_name

FROM fnd_conc_pp_actions fcpp, fnd_concurrent_processes fcp

WHERE fcpp.processor_id = fcp.concurrent_process_id

AND fcpp.action_type = 6

AND fcpp.concurrent_request_id = &&request_id

Note: The SQL statement will only return data for those Concurrent Requests having a Layout specified (Upon

Completion... window).

2. Via the Forms application:

• 1.Login to the application as SYSADMIN

• 2. Responsibility: System Administrator

• 3. Function: Concurrent --> Manager --> Administration

• 4. Select the Output Post Processor

• 5. Click on the Processes button

• 6. Select the Concurrent Process which was active during the time that the request ran

• 7. Click on the Manager Log button to open the Output Post Processor log file


3. the Oracle Application Manager (OAM)

• 1. Login to the application as SYSADMIN

• 2. Responsibility: System Administration

• 3. Function: Oracle Applications Manager --> Concurrent Managers

• 4. Select the Output Post Processor Service and click on View Details

• 5. Click on View Processes

• 6. Select the Concurrent Process which was active during the time that the request ran

• 7. Click on the Log button to open the Output Post Processor log file


XML Publisher Core Engine Debug logs

The XDO log consist of complete debug information generated by XML Publisher Core engine. Follow the steps given

below to enable debug

1. Connect to the concurrent manager server as Application Owner(applmgr).

2. Create an $XDO_TOP/temp and an $XDO_TOP/resource directory.

3. Create an xdodebug.cfg file in the $XDO_TOP/resource directory, containing the following 2 lines:

LogLevel=STATEMENT

LogDir=[full XDO_TOP]/temp

4. Restart the concurrent managers in case of single step publishing method (not required for two step publishing method).

5. Reproduce the problem.

Note: The xdodebug.cfg file can also be created in the $AF_JRE_TOP/jre/lib OR $AF_JRE_TOP/lib directory.

With the above steps several debug files will be created under the $XDO_TOP/temp directory after reproducing the

Problem

xdo.log : XML Publisher Core engine debug log file

xdo_...xsl : XSL-FO version of the RTF template

xdo_...xml : XML data file

xdo_...fo : temporary file created by the FO Engine (only RTF / XSL templates)

xdo_...out : output file (may not exist in case of complete failure).


Troubleshooting BI Publisher Issues

Out Of Memory Error :

Symptoms: The Report completes with an Error The log file includes a message similar to the following:

+------------- 1) PUBLISH -------------+

Beginning post-processing of request 744134 on node R12P660 at 10-JAN-2012

14:50:47. Post-processing of request 744134 failed at 10-JAN-2012 14:51:15 with the

Error message: One or more post-processing actions failed. Consult the OPP service log for

details.

OR

'****Warning!!! Due to high volume of data, got out of memory exception...***

'****Please retry with scalable option or modify the Data template to run in scalable mode...***

OR

java.lang.OutOfMemoryError: Java heap space

Memory Issues can be at 2 levels:

1)Xml Data generation

2)Report Generation from XML data


Xml Data Generation

If the XML data generation is not proper, follow the steps given below

Steps:

Increase the memory from CP Definition

• Go to "System Administrator" responsibility.

• Navigate to Concurrent - > Program - > Define.

• Search for the report in the Concurrent Program (CP) window.

• Set the Options field to –Xmx2048M (Note: Don’t miss the dash before the X)

• Press Save

• Bounce the Concurrent Manager (optional).


Report Generation from XML data

XML Generation is fine but OPP fails with java.lang.OutOfMemoryError exception.

1.Check the current OPP JVM Heap size using following SQL query

SQL> select DEVELOPER_PARAMETERS from FND_CP_SERVICES where SERVICE_ID = (select

MANAGER_TYPE from FND_CONCURRENT_QUEUES where CONCURRENT_QUEUE_NAME = 'FNDCPOPP'

2. The default should be:

J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx512m

3. Increase the Heap Space per Process to 1024:

SQL> update FND_CP_SERVICES

set DEVELOPER_PARAMETERS = 'J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx1024m'

where SERVICE_ID =

(select MANAGER_TYPE

from FND_CONCURRENT_QUEUES

where CONCURRENT_QUEUE_NAME = 'FNDCPOPP');

SQL>Commit;

You need to restart the Concurrent Manager to make it effective. The maximim heap size should not

exceed 2048.


Bi Publisher Enhancement Parameters

Temporary Directory : Set this to a valid path, make sure that the temp directory is at least 5GB or 20x larger

than largest data file running. The application user should have read & write permissions to this directory

Navigation Path : XML Publisher Administrator responsibility

-> Choose Configuration tab

->General

->Temporary directory

There are few FoProcessing Parameters which can be fine tuned for better performance

Use XML Publisher's XSLT processor : Set this value to True

Enable scalable feature of XSLT processor : Set this value to False at Site level. For large reports set this value to true at data

definition level and template level.

Enable XSLT runtime optimization : Set this value to True


Specific Situations

There are situations for which the concurrent manager process (FNDLIBR) does produce an XML data file but the post-processing of

the request fails due to a different issue encountered, some of which the OPP does not even get invoked in the first place:

1. The Output Post Processor is not running or does not pick up the request:

... +------------- 1) PUBLISH -------------+

Unable to find an Output Post Processor service to post-process request 260006.

Check that the Output Post Processor service is running.

+--------------------------------------+ ...

The OPP log file cannot be determined using the SQL statement described earlier, as the OPP did not perform any action of the given

request.

Cause: OPP Manger is Down/Deactivated.

Action: Verify that the Output Post Processor is enabled and active

1. Profile Option "Concurrent: GSM Enabled" must be set to Y

2. Navigate to: Concurrent -> Managers -> Define

3. Query for Manager = 'Output Post Processor’ or Short Name = FNDCPOPP

4. Check the checkbox "Enable", save the changes.

5. Navigate to: Concurrent -> Managers -> Administer

6. Make sure the Output Post Processor Status is not 'Deactivated',Set Status to 'Activate' if necessary.

7. Bounce the concurrent manager for the changes to get reflected.


2. The Output Post Processor is running but has not picked up the request:

...

+------------- 1) PUBLISH -------------+

The Output Post-processor is running but has not picked up this request.

No further attempts will be made to post-process this request, and the request will be marked

with Warning status.

+--------------------------------------+

The OPP log file cannot be determined using the SQL statement described earlier, as the OPP did not perform

any action of the given request.

Cause: The number of concurrent requests that the Output Post Processor can handle in parallel depends upon the number of

Processes and the number of Threads Per Process. The default values are 2 Processes and 5 Threads Per Process so a total of

10 reports can be processed in parallel. In case there are other concurrent requests running which have already invoked the

OPP then it might happen that no additional requests can be picked up for a period of time.

Action:

1.Logon to Applications with "System Administrator" responsibility

2.Navigate to Concurrent -> Manager -> Define

3.Query for "Output Post Processor"

4.Click on "Work Shifts" and Increase the number of increase the number threads from 5 to 10

oracle.apps.fnd.cp.opp.OPPServiceThread:2:0:max_threads=5

to

oracle.apps.fnd.cp.opp.OPPServiceThread:2:0:max_threads=10

5.Bounce the concurrent Manager for the changes to get reflected.

Reference document for Process Vs Thread

Doc ID 1399454.1- Tuning Output Post Processor (OPP) to Improve Performance


3. The concurrent manager has timed out waiting for the Output Post-processor to finish this request.

...

+------------- 1) PUBLISH -------------+

The concurrent manager has timed out waiting for the Output Post-processor to finish this request.

Check that there are enough Output Post-processor service processes running.

More information may be found in the service process logfile.

+--------------------------------------+

The concurrent manager process has successfully invoked the OPP but it a timeout is encountered as the OPP takes too long to

complete the job

Cause: Once the Output Post Processor picks up the request, the BI Publisher engine is invoked to generate the final output file. The

time that this takes will depends on various elements such as size of the XML data file, complexity of the template, performance of

the server.By default a timeout will occur if it takes longer then 300 seconds (5 min.) for the BI Publisher engine to generate the

output file. The concurrent request will complete with status Warning and the request log file will contain above error Message.

Action Increase the value of the Concurrent:OPP Process Timeout as follows:

1.Logon to Applications with "System Administrator" responsibility

Navigate to Profile -> System

2.Query for "Concurrent:OPP Process Timeout".

Set the value to 10800 sec (3 hours)

3. Query for "Concurrent:OPP Response Timeout".

If the value of the Concurrent:OPP Response Timeout is 120 increase it to 240. (Value * 2)


Please provide the following information while raising SR with oracle support for XML Publisher Reports

1.Request log file of the failed concurrent program.

2.Out Post Processer log File


March 2012 : EBS Patching Overview & Troubleshooting o EMEA-Session : 13th March 2012 at 10:00 CET (GMT +2)

o US-Session : 14th March 2012 at 18:00 CET (GMT +2)

For complete details on all upcoming Oracle Advisor Webcast Events, please see

Note 740966.1, Oracle Advisor Webcast Schedule.

For EBS Technology Specific Webcasts please check Note 1186338.1.

Do you have any requests for future ATG Advisor Webcast Events ?

Please email your suggestions to me :

[email protected], subject: Topics of Interest.

E-Business Suite Technology Group - Webcasts

https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=740966.1



Communities in My Oracle Support


EBS Technology Communities in MOS available Following Communities are available in My Oracle Support

Oracle E-Business Suite

...

BI Publisher - Business Intelligence Products (not only EBS)

Core Concurrent Processing - Anything around Concurrent Processing and Concurrent Managers

Core Workflow - Any Workflow isse not only E-Business Suite

Diagnostic Tools - Anything around EBS Diagnostics

E-Business Customizations - Your Customizations

Installation - Fresh Install of the E-Business Suite

Patch Review EBS - Review of Patches around the E-Business Suite

Performance - EBS Performance

Upgrade - EBS Upgrade

User Produktivity Kit - User Productivity Kit (UPK) available for the E-Business Suite

Utilities - Utilities / generic EBS DBA issues

...

This is the current list for the E-Business Suite – Applications Technology Group

https://communities.oracle.com/portal/server.pt/community/bi_publisher/323

https://communities.oracle.com/portal/server.pt/community/core_concurrent_processing/493

https://communities.oracle.com/portal/server.pt/community/core_workflow/244

https://communities.oracle.com/portal/server.pt/community/diagnostic_tools/242

https://communities.oracle.com/portal/server.pt/community/e-business_customizations/322



https://communities.oracle.com/portal/server.pt/community/installation/257

https://communities.oracle.com/portal/server.pt/community/patch_reviews_-_ebs/358

https://communities.oracle.com/portal/server.pt/community/performance/333

https://communities.oracle.com/portal/server.pt/community/upgrade/258

https://communities.oracle.com/portal/server.pt/community/user_productivity_kit/235

https://communities.oracle.com/portal/server.pt/community/utilities/259


EBS ATG Product Information Center Note 1160285.1



Q&A


THANK YOU

for attending our Advisor Webcast!

2012 Feb EBS Concurrent Processing Troubleshooting

Documents

Transcript of 2012 Feb EBS Concurrent Processing Troubleshooting