Blocks Corrupted in system09.dbf After Fresh Installation of E-Business Suite R12.1.1

We have identified a block corruption in the 12.1.1 Fresh Installed Instance, This block corruption identified while running RMAN.

RMAN LOG:
=========

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 09/06/2011 19:51:02
ORA-19566: exceeded limit of 0 corrupt blocks for file /u01/db/apps_st/data/system09.dbf

SQL> select * from V$DATABASE_BLOCK_CORRUPTION;
FILE# BLOCK# BLOCKS CORRUPTION_CHANGE# CORRUPTIO
---------- ---------- ---------- ------------------ ---------
352 165657 1 0 ALL ZERO
352 165656 1 0 CORRUPT

SQL> select file_name,FILE_ID,TABLESPACE_NAME from dba_data_files where file_id=352;
FILE_NAME FILE_ID TABLESPACE_NAME
------------------------------------------------------------ ---------- ------------------------------
/u01/db/apps_st/data/system09.dbf 352 SYSTEM

DB Verify OUTPUT
================
dbv file=/u01/db/apps_st/data/system09.dbf feedback=1000
DBVERIFY - Verification complete
Total Pages Examined : 192128
Total Pages Processed (Data) : 53193
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 27615
Total Pages Failing (Index): 0
Total Pages Processed (Other): 2081
Total Pages Processed (Seg) : 0
Total Pages Failing (Seg) : 0
Total Pages Empty : 82766
Total Pages Marked Corrupt : 26473
Total Pages Influx : 0
Total Pages Encrypted : 0
Highest block SCN : 3723391595 (1388.3723391595)


The Segments are Free segments - No need to Panic
=====================================
SQL> SELECT e.owner, e.segment_type, e.segment_name, e.partition_name, c.file#
2 , greatest(e.block_id, c.block#) corr_start_block#
, least(e.block_id+e.blocks-1, c.block#+c.blocks-1) corr_end_block#
3 , least(e.block_id+e.blocks-1, c.block#+c.blocks-1)
4 5 - greatest(e.block_id, c.block#) + 1 blocks_corrupted
, null description
6 7 FROM dba_extents e, v$database_block_corruption c
8 WHERE e.file_id = c.file#
9 AND e.block_id <= c.block# + c.blocks - 1
10 AND e.block_id + e.blocks - 1 >= c.block#
11 UNION
12 SELECT s.owner, s.segment_type, s.segment_name, s.partition_name, c.file#
13 , header_block corr_start_block#
14 , header_block corr_end_block#
15 , 1 blocks_corrupted
16 , 'Segment Header' description
17 FROM dba_segments s, v$database_block_corruption c
18 WHERE s.header_file = c.file#
19 AND s.header_block between c.block# and c.block# + c.blocks - 1
20 UNION
21 SELECT null owner, null segment_type, null segment_name, null partition_name, c.file#
22 , greatest(f.block_id, c.block#) corr_start_block#
23 , least(f.block_id+f.blocks-1, c.block#+c.blocks-1) corr_end_block#
24 , least(f.block_id+f.blocks-1, c.block#+c.blocks-1)
25 - greatest(f.block_id, c.block#) + 1 blocks_corrupted
, 'Free Block' description
26 27 FROM dba_free_space f, v$database_block_corruption c
28 WHERE f.file_id = c.file#
29 AND f.block_id <= c.block# + c.blocks - 1
AND f.block_id + f.blocks - 1 >= c.block#
30 31 order by file#, corr_start_block#;


OWNER SEGME SEGME PARTITION_NA FILE# CORR_START_BLOCK# CORR_END_BLOCK# BLOCKS_CORRUPTED DESCRIPTION
---------- ----- ----- ------------ ---------- ----------------- --------------- ---------------- --------------
352 165656 165656 1 Free Block
352 165657 165657 1 Free Block

SOLUTION TO FIX THE ISSUE
==========================

This is known issue in R12.1.1, The installation media of R12 has block corruption in it.

Ref NoteID: ID 840963.1

Have performed below actions to fix it.

SQL> select sum(bytes/1024/1024),sum(maxbytes/1024/1024) from dba_data_files where FILE_NAME like '%system09.dbf';
SUM(BYTES/1024/1024) SUM(MAXBYTES/1024/1024)
-------------------- -----------------------
                1501                       0

SQL>alter database datafile '/u01/db/apps_st/data/system09.dbf' resize 1000M;

Check for Corruption Now: NONE :)


$dbv file=/u01/db/apps_st/data/system09.dbf feedback=1000
DBVERIFY: Release 11.1.0.7.0 - Production on Wed Sep 7 08:41:53 2011
Copyright (c) 1982, 2007, Oracle.  All rights reserved.
DBVERIFY - Verification starting : FILE = /u01/db/apps_st/data/system09.dbf
................................................................................
................................................
DBVERIFY - Verification complete
Total Pages Examined         : 128000
Total Pages Processed (Data) : 53193
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 27615
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 2081
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 45111
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3723488883 (1388.3723488883)

Unable To View workflow Status Diagram

In a fresh R12 installed instance, When clicked on the Status diagram following message appears "You have encountered an unexpected error. Please contact the System Administrator for assistance."

When pressed on the click Here link the below error message displays:
oracle.apps.fnd.framework.OAException: java.lang.NullPointerException ( and it displays lot of lines with the exception message)

Solution:

Set the profile option Server Timezone at Site level (should be set to the value of database timezone) and Bounce the Apache Server

Function not available to this responsibility. Change Responsibilities or contact your System Administrator

Function not available to this responsibility. Change Responsibilities or contact your System Administrator

In R12 after setting up New CUSTOM TOP, When you try to access Forms we are hitting "Function not available to this responsibility. Change Responsibilities or contact your System Administrator"

Even after defining the CUSTOM_TOP properly and defining the TOP in application env, this error would occur.

Solution:

1)In R12 go to $INST_TOP/ora/10.1.2/forms/server
2)Make sure your CUSTOM_TOP is defined in  default.env
Example:
Add the entrey as below  in the default.env
XXXYZ=/u01/oracle/EBSR12/apps/apps_st/appl/xxxyz/12.0.0

3)Bounce adopmnctl.sh stopall & Startall

4)Re-test the issue.

What is REVIVER In Oracle Applications:

What is REVIVER In Oracle Applications:

Reviver is used for momentary disconnects in the system where the concurrent managers and/or forms go down, and forms is later reconnected while the concurrent managers are not. FNDREVIVER revives the Internal Concurrent Manager (ICM) when it fails.
When ICM can no longer get a database connection, it kills itself and spawns the reviver. Reviver loops every 30 seconds, attempting to login to the database as apps user. Once login is successful, it starts up the ICM again.

If the failure is due to a brief network outage, or database issue, the managers are restarted, so the client does not have to restart the managers manually.

Reviver is recovery mechanism delivered with ATG RUP4 (patch 4676589), and runs in the background. In a Real Applications Cluster (RAC) environment, when the primary node goes down and ICM is set to migrate to the secondary node, the reviver parameter will be passed to the secondary node.

The easiest way to determine if reviver.sh exists is by checking the $FND_TOP/bin directory.

The variable resides in the context file under 's_cp_reviver' and can be set to "enabled" or "disabled". Based on the value of s_cp_reviver in the context file, AFCPDNR is started with a value of either "enabled" or "disabled" .

The reviver is started when starting the ICM, by passing a parameter reviver="enabled". You do this on the node you start the manager, and if the ICM is set to migrate to the second node, this parameter will be passes to the second node.
A common misconception is that users must start the reviver.sh manually, however this is not the intended use. It is automatically enabled when the parameter REVIVER_PROCESS="enabled" is passed via the adcmctl.sh concurrent manager startup script.

On a single node concurrent processing system, FNDREVIVER is the only way to recover from a database connection loss.

On a two node system (PCP), there is another factor, the Internal Monitor (FNDIMON).The FNDIMON will race to restart the internal manager in a multi node setup, and by the time the reviver starts it will likely see that the ICM is already running and exit accordingly.
FNDIMON checks whether it can connect to the database in order to determine if the ICM is running, and if the database connection is not available it fails to run and exits accordingly. The reviver is a shell script which loops until a connection is obtained, and then starts the manager accordingly. The reviver's job is the last line of defense after a database connection failure, as FNDIMON only works when the database connection is available.

In the event the ICM goes down due to a network outage, then the reviver would be needed to bring the ICM back up.

How to check and Enable reviver in Oracle Applications (Its by default enabled in R12):

The following parameters can be set in the context file, and then autoconfig should be re-run to enable reviver:

Example:
 $grep -i s_cp_reviver $CONTEXT_FILE
<cp_reviver oa_var="s_cp_reviver">enabled</cp_reviver>

Reviver Process PID Directory Location (s_fndreviverpiddir)

This variable specifies the path where ICM reviver process pid file will be created. Oracle recommends using a local disk as the PID file location because
the reviver process may run when the network is down.

Example:
<fndreviverpiddir oa_var="s_fndreviverpiddir">/u02/oracle/visappl/fnd/11.5.0/log</fndreviverpiddir>

All about DataInstaller & Hrglobal.drv in Oracle Applications

All about DataInstaller & Hrglobal.drv in Oracle Applications

This posts covers what is datainstaller & hrglobal.drv and when do we need to run them.

DataInstaller and hrglobal.drv are used for installing legislative data into Oracle Applications,For example a company operates in multiple counntries and payroll and HR practices are different in each countires, So they need to have all the legislative regulations (Taxation and HR rules and regulations) of the all the countries they run their business.

To achive this Oracle provides a set of patches, which need to applied to the Oracle applications.

These patches are country specific, The patch matrix can be found from Latest Oracle HRMS Legislative Data Patch Available (HR Global / hrglobal) [ID 145837.1]

To Install the legislative data please follow the below 4 steps:

1) PER & PAY modules need to be licenced ( This is madantory requirement)

2) Identify the patches & apply them based on your requirement from the Note:  145837.1
(Check the countires you need legislations and versions of the Applications)

3)After applying the patches run the datainstaller

Datainstaller is java command which will launch a interface to choose the set of legislation you require.

Always Install Global along with the country legislation you install.

Ex: If you wish to install US  legislations , Install Global and US.

You can install either payroll or hr for the the specific country your require or you can choose both.

The command line for the Data Install program is as follows:

Always run the datainstaller command from Middle Tier (Not from DB node) command line (VNC not needed)

Syntax:
java oracle.apps.per.DataInstall apps <apps_pwd> thin <db Hostname>:<db port>:<Db SID>
Example:

java oracle.apps.per.DataInstall apps apps thin testdb.oracle.com:1521:TESTDB

You will see a screen like below:

DataInstall Main Menu

1. Select legislative data to install/upgrade
2. Select college data to install/upgrade
3. Select JIT/Geocode or OTL to install/upgrade
4. Exit to confirmation menu

Enter your choice : 1

For more details on how to run Data Installer and hrglobal.drv Please check NOTE ID 140511.1

4)The last step in this process is to apply the hrglobal.drv

hrglobal.drv is patch driver file located in $PER_TOP/patch/115/driver, Apply it using
adpatch to populate the legislative data into the required HR tables in the database.

hglobal.drv can be applied as hotpatch.

Syntax:
adpatch driver=$PER_TOP/patch/115/driver/hrglobal.drv logfile=hrglobal.drv.log options=hotpatch <optional>

FAQ

Q) Do we need to run the datainstaller and hrglobal.drv every time we apply a HRMS patch?

A) No, Not needed, You need to run datainstaller when ever you apply the HRMS RUP patches or Upgrade Oracle Apps or Upgrade HRMS module.

In the above cases you need re0check the patch matrix and apply the nessesaty paches and run datinstaller & hrglobal.drv.

If any patch explicitely says you to run the datainstaller & hrglobal.drv you need to run them

Basically for regular HRMS patching (one-offs and Mini) you no need to run the datainstaller.

Sometimes you may need to apply hrglobal.drv and you may not needed datainstaller (when there is no change in the legislative selection)