Autoconfig On Db Tier Fails With Error – SP2-1503: Unable to initialize Oracle call interface

Oracle Applications 11.5.10.2 on Solaris 5.10 64bit SPARC

Oracle Application Database 10gR2 (10.2.0.4) upgraded to Oracle Database 11.2.0.3 using Interoperability Notes Oracle EBS 11i with Oracle Database 11gR2 (11.2.0) [ID 881505.1].

Issue encountered while executing adconfig in the end:

$>./adconfig.sh
Enter the full path to the Context file: /oracle01/myinstance/r12/db/product/11.2.0/dbhome_1/appsutil/myinstance_myserver.xml
Enter the APPS user password:
The log file for this session is located at: /oracle01/myinstance/r12/db/product/11.2.0/dbhome_1/appsutil/log/myinstance_myserver/07300952/adconfig.log

AutoConfig is configuring the Database environment...

AutoConfig will consider the custom templates if present.
Using ORACLE_HOME location : /oracle01/myinstance/r12/db/product/11.2.0/dbhome_1
Classpath                   : /oracle01/myinstance/r12/db/product/11.2.0/dbhome_1/jdk/jre/lib/rt.jar:/oracle01/myinstance/r12/db/product/11.2.0/dbhome_1/jdk/lib/dt.jar:/oracle01/myinstance/r12/db/product/11.2.0/dbhome_1/jdk/lib/tools.jar:/oracle01/myinstance/r12/db/product/11.2.0/dbhome_1/jdbc/lib/ojdbc5.jar:/oracle01/myinstance/r12/db/product/11.2.0/dbhome_1/appsutil/java/xmlparserv2.zip:/oracle01/myinstance/r12/db/product/11.2.0/dbhome_1/appsutil/java:/oracle01/myinstance/r12/db/product/11.2.0/dbhome_1/jlib/netcfg.jar

Using Context file          : /oracle01/myinstance/r12/db/product/11.2.0/dbhome_1/appsutil/myinstance_myserver.xml

Context Value Management will now update the Context file

Updating Context file...COMPLETED

Attempting upload of Context file and templates to database...COMPLETED

Updating rdbms version in Context file to db112
Updating rdbms type in Context file to 64 bits
Configuring templates from ORACLE_HOME ...

AutoConfig completed with errors.

adconfig.log
============

[AutoConfig Error Report]
The following report lists errors AutoConfig encountered during each
phase of its execution.  Errors are grouped by directory and phase.
The report format is:
<filename>  <phase>  <return code where appropriate>

[PROFILE PHASE]
AutoConfig could not successfully execute the following scripts:
Directory: /oracle01/myinstance/r12/db/product/11.2.0/dbhome_1/appsutil/install/myinstance_myserver
afdbprf.sh              INSTE8_PRF         1

[APPLY PHASE]
AutoConfig could not successfully execute the following scripts:
Directory: /oracle01/myinstance/r12/db/product/11.2.0/dbhome_1/appsutil/install/myinstance_myserver
adcrobj.sh              INSTE8_APPLY       1
AutoConfig is exiting with status 2

Cause:
SP2-1503: Unable to initialize Oracle call interface
SP2-0152: ORACLE may not be functioning properly

Solution:

The issue is related to ORA_TZFILE env variable in 11GR2 this env variable is no longer needed.

In Oracle Database server to fix this issue Unset the env varaible ORA_TZFILE and try to connect.

In Oracle Applications Database to fix this issue Perform the below.

The ORA_TZFILE is set while at 11gR2 this should not be set.

The  $ORACLE_HOME/appsutil/template/ad8idbux.env is from the wrong version  (ad8idbux.env 120.8)

The wrong file includes the  following:

ORA_TZFILE="$ORACLE_HOME/oracore/zoneinfo/timezlrg.dat"

export  ORA_TZFILE

While for 11gr2 it should be:

if test "%s_database%" =  "db112" ;  then

ORA_TZFILE=""

else

ORA_TZFILE="$ORACLE_HOME/oracore/zoneinfo/timezlrg.dat"

fi

export  ORA_TZFILE

Source the environment variable and executed autoconfig in Database Node, completed without issue

Reference: Autoconfig On Db Tier Fails With Error – SP2-1503: Unable to initialize Oracle call interface (Doc ID 1187616.1)

The issue is also resolved when you upgrade the applications 11i to Release 12 (12.1.1) or higher version on 11gR2 Database.

Advertisements

ORA-15020: discovered duplicate ASM disk

I was installing Grid infrastructure for a standalone server.

By default, the diskstring was ORCL:* but I changed it to /dev/oracleasm/disks. However, ths disks were visible on both diskstrings. After the change, I faced the following error message at the end of installation:

INFO: Command /u01/app/grid/product/11.2.0/grid/bin/asmca -silent -oui_internal -configureASM -diskString /dev/oracleasm/disks -diskGroupName CRS -diskList /dev/oracleasm/disks/CRSVOL1,/dev/oracleasm/disks/CRSVOL2,/dev/oracleasm/disks/CRSVOL3 -redundancy EXTERNAL
INFO: ... GenericInternalPlugIn.handleProcess() entered.
INFO: ... GenericInternalPlugIn: getting configAssistantParmas.
INFO: ... GenericInternalPlugIn: checking secretArguments.
INFO: ... GenericInternalPlugIn: starting read loop.
INFO: Read: SYS_PASSWORD_PROMPT
INFO: Processing: SYS_PASSWORD_PROMPT for argument tag -sysAsmPassword
INFO: Read: ASMSNMP_PASSWORD_PROMPT
INFO: Processing: ASMSNMP_PASSWORD_PROMPT for argument tag -asmMonitorPassword
INFO: End of argument passing to stdin
INFO: Read:
INFO: Read: Configuring ASM failed with the following message:
INFO: Read: One or more diskgroup(s) creation failed as below:
INFO: Read: DiskGroup CRS creation failed with the following message:
INFO: Read: ORA-15018: diskgroup cannot be created
INFO: Read: ORA-15020: discovered duplicate ASM disk "CRS_0002"
INFO: Read:
INFO: Read:
INFO: Completed Plugin named: Automatic Storage Management Configuration Assistant

Solution:

I changed the diskstring to ORCL as shown below and ran the below command as grid user

 /u01/app/grid/product/11.2.0/grid/bin/asmca -silent -oui_internal -configureASM -diskString ORCL:* -diskGroupName CRS -diskList ORCL:CRSVOL1,ORCL:CRSVOL2,ORCL:CRSVOL3 -redundancy EXTERNAL

If the ASM configuration assitant fails during the installation, then you can run the command manually after changing the diskstring and skip the step on the GUI installer after the command completes successfully.

In my case, the corrected command ran successfully.

DNS configuration for the SCAN used with Oracle Database 11gR2

This post provides the basic DNS configuration steps necessary to use the Single Client Access Name (SCAN) introduced in Oracle 11g Release 2 RAC.

I’m not an network guy and don’t claim to know much about DNS.

This post represents the minimum you can do to make a SCAN work. The DNS configuration in the post has been done on a DNS server installed on a separate VM called romford.mpoojari.co.uk where romford is the hostname (dns server name) and mpoojari.co.uk is the domain name. The IP address of this server will be 192.168.0.10 (static IP).

DNS installation

Install and configure RHEL 5.4 VM

[root@romford etc]# uname -a
Linux romford.mpoojari.co.uk 2.6.18-194.el5 #1 SMP Mon Mar 29 22:10:29 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

Ensure you have choose DNS Name server and Server configuration tools(select bind packages) during the installation phase.

Check if the following DNS Server Packages are installed

[root@romford ~]# rpm -qa | grep bind
bind-libs-9.3.6-4.P1.el5_4.2
ypbind-1.19-12.el5
system-config-bind-4.0.3-4.0.1.el5
bind-utils-9.3.6-4.P1.el5_4.2
bind-chroot-9.3.6-4.P1.el5_4.2
bind-9.3.6-4.P1.el5_4.2

Three packages must be installed on Linux for DNS Server:
•bind (includes DNS server, named)
•bind-utils (utilities for querying DNS servers about host information)
•bind-libs (libraries used by the bind server and utils package)

You should be able to locate /etc/named.conf
If not copy  all the files under /usr/share/doc/bind-9.3.4/sample/etc  to /etc
Also copy all the files under /usr/share/doc/bind-9.3.4/sample/var  to /var

[root@romford etc]# ls -ltr /usr/share/doc/bind-9.3.6/sample/etc
total 32
-rw-r--r-- 1 root root  524 Jan 20  2010 named.root.hints
-rw-r--r-- 1 root root  775 Jan 20  2010 named.rfc1912.zones
-rw-r--r-- 1 root root 4298 Jan 20  2010 named.conf
-rw-r--r-- 1 root root    0 Jan 20  2010 rndc.conf

[root@romford etc]# ls -ltr /usr/share/doc/bind-9.3.6/sample/var/
total 8
drwxr-xr-x 4 root root 4096 Nov 24 14:01 named
[root@romford etc]# ls -ltr /usr/share/doc/bind-9.3.6/sample/var/named/
total 88
-rw-r--r-- 1 root root  427 Jan 20  2010 named.zero
-rw-r--r-- 1 root root 1892 Jan 20  2010 named.root
-rw-r--r-- 1 root root  426 Jan 20  2010 named.local
-rw-r--r-- 1 root root  424 Jan 20  2010 named.ip6.local
-rw-r--r-- 1 root root  427 Jan 20  2010 named.broadcast
-rw-r--r-- 1 root root  195 Jan 20  2010 localhost.zone
-rw-r--r-- 1 root root  198 Jan 20  2010 localdomain.zone
-rw-r--r-- 1 root root   56 Jan 20  2010 my.internal.zone.db
-rw-r--r-- 1 root root   56 Jan 20  2010 my.external.zone.db
drwxr-xr-x 2 root root 4096 Jan 20  2010 data
drwxr-xr-x 2 root root 4096 Nov 24 14:01 slaves

1. Generate the DNS Key

[root@romford etc]# which dns-keygen
/usr/sbin/dns-keygen

[root@romford etc]# dns-keygen
j30zdeKhACTGFWSdxGgHVSC3jFm0ejIAn5p4mJkYmA0cyHp4jDaT30SYQLMR

2. Configure the named.conf file

Add the DNS key to the /etc/named.conf under the below section and should look like

replace

key ddns_key
{
        algorithm hmac-md5;
        secret "use /usr/sbin/dns-keygen to generate TSIG keys";
};

to

key ddns_key
{
         algorithm hmac-md5;
         secret "j30zdeKhACTGFWSdxGgHVSC3jFm0ejIAn5p4mJkYmA0cyHp4jDaT30SYQLMR";
 };

Add the below line in the /etc/named.conf .

zone "mpoojari.co.uk" IN {
 type master;
 file "mpoojari.co.uk.zone";
 allow-update { none; };
};

3. Create the zone file under /var/named

[root@romford etc]# ls -ltr /var/named
total 44
drwxr-xr-x 2 root root 4096 Nov 24 16:27 slaves
-rw-r--r-- 1 root root  427 Nov 24 16:27 named.zero
-rw-r--r-- 1 root root 1892 Nov 24 16:27 named.root
-rw-r--r-- 1 root root  426 Nov 24 16:27 named.local
-rw-r--r-- 1 root root  424 Nov 24 16:27 named.ip6.local
-rw-r--r-- 1 root root  427 Nov 24 16:27 named.broadcast
-rw-r--r-- 1 root root   56 Nov 24 16:27 my.internal.zone.db
-rw-r--r-- 1 root root   56 Nov 24 16:27 my.external.zone.db
-rw-r--r-- 1 root root  195 Nov 24 16:27 localhost.zone
-rw-r--r-- 1 root root  198 Nov 24 16:27 localdomain.zone
drwxr-xr-x 2 root root 4096 Nov 24 16:27 data

[root@romford etc]# cd /var/named
[root@romford named]# cp localdomain.zone mpoojari.co.uk.zone
[root@romford named]# ls -ltr
total 48
drwxr-xr-x 2 root root 4096 Nov 24 16:27 slaves
-rw-r--r-- 1 root root  427 Nov 24 16:27 named.zero
-rw-r--r-- 1 root root 1892 Nov 24 16:27 named.root
-rw-r--r-- 1 root root  426 Nov 24 16:27 named.local
-rw-r--r-- 1 root root  424 Nov 24 16:27 named.ip6.local
-rw-r--r-- 1 root root  427 Nov 24 16:27 named.broadcast
-rw-r--r-- 1 root root   56 Nov 24 16:27 my.internal.zone.db
-rw-r--r-- 1 root root   56 Nov 24 16:27 my.external.zone.db
-rw-r--r-- 1 root root  195 Nov 24 16:27 localhost.zone
-rw-r--r-- 1 root root  198 Nov 24 16:27 localdomain.zone
drwxr-xr-x 2 root root 4096 Nov 24 16:27 data
-rw-r--r-- 1 root root  198 Nov 24 16:33 mpoojari.co.uk.zone

Add the below entries to the zone file /var/named/mpoojari.co.uk.zone (and /var/named/localdomain.zone file not required as my domain is mpoojari.co.uk)

 
rac1            IN A            192.168.0.101
rac2            IN A            192.168.0.102
rac1-vip        IN A            192.168.0.221
rac2-vip        IN A            192.168.0.222
rac1-priv       IN A            192.168.1.101
rac2-priv       IN A            192.168.1.102
rac-scan        IN A            192.168.0.51
rac-scan        IN A            192.168.0.52
rac-scan        IN A            192.168.0.53

4. Configure a reverse proxy
To identify the SCAN name from any of the IPS’s we need to configure a reverse proxy .
Add the below entry to /etc/named.conf

zone "0.168.192.in-addr.arpa." IN {
 type master;
 file "0.168.192.in-addr.arpa";
 allow-update { none; };
 };

5. Create the reverse proxy file under /var/named

 
[root@ast1 named]# cat 0.168.192.in-addr.arpa
$ORIGIN 0.168.192.in-addr.arpa.
$TTL 1H
@       IN      SOA     mpoojari.co.uk.     root.mpoojari.co.uk. (      2
                                                3H
                                                1H
                                                1W
                                                1H )
0.168.192.in-addr.arpa.         IN NS      mpoojari.co.uk.

101     IN PTR  rac1.mpoojari.co.uk.
102     IN PTR  rac2.mpoojari.co.uk.
221     IN PTR  rac1-vip.mpoojari.co.uk.
222     IN PTR  rac2-vip.mpoojari.co.uk.
51      IN PTR  rac-scan.mpoojari.co.uk.
52      IN PTR  rac-scan.mpoojari.co.uk.
53      IN PTR  rac-scan.mpoojari.co.uk.

The named.conf file finally looks like as shown below

[root@romford etc]# cat /etc/named.conf
options
{
        // Those options should be used carefully because they disable port
        // randomization
        // query-source    port 53;
        // query-source-v6 port 53;

        // Set IP address correctly.
        listen-on port 53 { 127.0.0.1; 192.168.0.10; };

        // My Additions
        // Forwarder: Anything this DNS can't resolve gets forwarded to my ISPs DNS.
        // forwarders { 194.168.4.100; 194.168.8.100; };
        // End My Additions

        // Put files that named is allowed to write in the data/ directory:
        directory "/var/named"; // the default
        dump-file               "data/cache_dump.db";
        statistics-file         "data/named_stats.txt";
        memstatistics-file      "data/named_mem_stats.txt";

};
zone "mpoojari.co.uk" IN {
        type master;
        file "mpoojari.co.uk.zone";
        allow-update { none; };
};
zone "0.168.192.in-addr.arpa." IN {
        type master;
        file "0.168.192.in-addr.arpa";
        allow-update { none; };
};
key ddns_key
{
        algorithm hmac-md5;
        secret "j30zdeKhACTGFWSdxGgHVSC3jFm0ejIAn5p4mJkYmA0cyHp4jDaT30SYQLMR";
};

6. Restart the named service as below

[root@romford named]# service named stop
Stopping named:                                            [  OK  ]

[root@romford named]# service named start
Starting named:                                            [  OK  ]

7. Edit the /etc/resolv.conf and put your domain name and the DNS server name on all nodes

[root@romford named]# cat /etc/resolv.conf
search localdomain mpoojari.co.uk
nameserver 192.168.0.10

8. On the DNS server

[root@romford named]# cat /etc/nsswitch.conf |grep dns
#       dns                     Use DNS (Domain Name Service)
#hosts:     db files nisplus nis dns
hosts:      dns files nis

9. Do a nslookup on SCAN and observe the IP addresses round-robin resolution

[root@romford etc]# nslookup rac-scan.mpoojari.co.uk
Server:         192.168.0.10
Address:        192.168.0.10#53

Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.52
Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.53
Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.51

[root@romford etc]# nslookup rac-scan.mpoojari.co.uk
Server:         192.168.0.10
Address:        192.168.0.10#53

Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.51
Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.52
Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.53

[root@romford etc]# nslookup rac-scan.mpoojari.co.uk
Server:         192.168.0.10
Address:        192.168.0.10#53

Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.53
Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.51
Name:   rac-scan.mpoojari.co.uk
Address: 192.168.0.52

If try to ping rac-scan.testenv.com at this moment, you will find it displays one of SCAN ip addresses but it will not be able to be reached. This is correct behavior. Also at this stage, I have not installed rac1 and rac2 nodes hence they will not be able to be reached.

Once the GI software has been installed and is running it will bring these IP addresses online and at that point the SCAN IP should be pingable.

10. nslookup on VIP names

[root@romford etc]# nslookup rac1-vip
Server:         192.168.0.10
Address:        192.168.0.10#53

Name:   rac1-vip.mpoojari.co.uk
Address: 192.168.0.221

[root@romford etc]# nslookup rac2-vip
Server:         192.168.0.10
Address:        192.168.0.10#53

Name:   rac2-vip.mpoojari.co.uk
Address: 192.168.0.222

Troubleshooting

Error while restarting named service,

[root@romford etc]# service named restart
Stopping named:                                            [  OK  ]
Locating /var/named/chroot//etc/named.conf failed:
                                                           [FAILED]

I’ve commented, “#ROOTDIR=/var/named/chroot” in /etc/sysconfig/named.
And now service named will start!

[root@romford etc]# service named restart
Stopping named:                                            [  OK  ]
Starting named:                                            [  OK  ]

Network Configuration of rac nodes

Node1

  • Set DNS server ip address to 192.168.0.10
  • Set gateway ip address to 192.168.0.1
  • Set hostname as rac1.mpoojari.co.uk
  • set static IP to 192.168.0.101 (eth0)
  • set static IP to 192.168.1.101 (eth1)
  • Set subnet mask 255.255.255.0

Node2

  • Set DNS server ip address to 192.168.0.10
  • Set gateway ip address to 192.168.0.1
  • Set hostname as rac2.mpoojari.co.uk
  • set static IP to 192.168.0.102 (eth0)
  • set static IP to 192.168.1.102(eth1)
  • Set subnet mask 255.255.255.0

At this stage,

rac1 is reachable from DNS server, romford and from rac2
rac2 is reachable from DNS server, romford and from rac1

SCAN name is also pingable on rac1 and rac2.

Note that hosts files  do not contain any IP addresses as it is DNS resolved.

RMAN-06059: expected archived log not found

RMAN-06059: expected archived log not found

RMAN attempted to backup an archive log file, but couldn’t find it.

This can happen for a variety of reasons; the file has been manually moved or deleted, the archive log destination has recently been changed, the file has been compressed, etc.

Starting backup at 26-JUL-12
current log archived
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=155 devtype=DISK
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 07/26/2012 22:55:32
RMAN-06059: expected archived log not found, lost of archived log compromises recoverability
ORA-19625: error identifying file /app/oracle/product/10.2/dbs/arch1_1_781981525.dbf
ORA-27037: unable to obtain file status
SVR4 Error: 2: No such file or directory
Additional information: 3

Your options are either to restore the missing file(s), or to perform a crosscheck. To perform a crosscheck, run the following command from within RMAN:

change archivelog all crosscheck;
It is advisable to perform a full backup of the database at this point.

When an archive log crosscheck is performed, RMAN checks each archive log in turn to make sure that it exists on disk (or tape). Those that are missing are marked as unavailable. If you have got missing logs, this won’t bring them back. It will allow you to get past this error and back-up the database though.

ERROR conn.ConnectionService verifyRepositoryEx.433 – Invalid Connection Pool. ERROR = ORA-28000: the account is locked

Issue below can occur on any platform on Oracle Database 10g and 11g database console.

Oracle Database Console started up without any error messages. EM Database console upon access showed database status down and listener status up. On reviewing the sysman log file found the following error:

2012-05-21 17:25:50,426 [HttpRequestHandler-15749845] ERROR

conn.ConnectionService verifyRepositoryEx.433 - Invalid Connection
Pool. ERROR = ORA-28000: the account is locked

Checked the database for SYSMAN

SQL> select username,account_status from dba_users where username in
('SYSMAN','DBSNMP');

USERNAME                       ACCOUNT_STATUS
------------------------------ ------------------------------
DBSNMP                         OPEN 
SYSMAN                         LOCKED(TIMED)

Password for SYSMAN was unknown for me at this stage and the configuration files had encrypted password in it.

Stopped the em console using

emctl stop dbconsole

Then on the database, changed the SYSMAN password as below:

SQL> alter user sysman identified by password account unlock;

User altered.

targets.xml

bash-3.00$ cd $ORACLE_HOME/<hostname_sid>/sysman/emd
bash-3.00$ ls -ltr targets.xml
-rw-r----- 1 oracle oinstall 894 May 21 17:25 targets.xml

bash-3.00$ cp targets.xml targets.xml.21052012

—Change from

<Property NAME="UserName" VALUE="ea8d8c4d8bb0ec12" ENCRYPTED="TRUE"/>
<Property NAME="password" VALUE="21a4a57992210188" ENCRYPTED="TRUE"/>

—to

<Property NAME="UserName" VALUE="SYSMAN" ENCRYPTED="FALSE"/>
<Property NAME="password" VALUE="password" ENCRYPTED="FALSE"/>

emoms.properties

bash-3.00$ cd $ORACLE_HOME/<hostname_sid>/config
bash-3.00$ cp emoms.properties emoms.properties.21052012

—Change from

oracle.sysman.eml.mntr.emdRepPwd=21a4a57992210188
oracle.sysman.eml.mntr.emdRepPwdEncrypted=TRUE

—to

oracle.sysman.eml.mntr.emdRepPwd=password
oracle.sysman.eml.mntr.emdRepPwdEncrypted=FALSE

During the DB console startup the password and username will get encrypted in the above 2 files.

bash-3.00$ emctl start dbconsole

Now while accessing the DB console showed up the login page and the new SYSMAN credentials worked!

ld.so.1: sqlplus: fatal: libsqlplus.so

Applies to Oracle Database 10g and higher versions, any platform.

The following settings are for Solaris 10 64-bit Operating system.

Cannot invoke sqlplus at the command line due to following library error

%sqlplus
ld.so.1: sqlplus: fatal: libsqlplus.so: open failed: No such file or directory
Killed

Print shared library dependencies for sqlplus

%ldd sqlplus
        libsqlplus.so =>         (file not found)
        libclntsh.so.10.1 =>     (file not found)
        libnnz10.so =>   (file not found)
        libkstat.so.1 =>         /lib/64/libkstat.so.1
        libnsl.so.1 =>   /lib/64/libnsl.so.1
        libsocket.so.1 =>        /lib/64/libsocket.so.1
        libgen.so.1 =>   /lib/64/libgen.so.1
        libdl.so.1 =>    /lib/64/libdl.so.1
        libsched.so.1 =>         /usr/lib/64/libsched.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        libaio.so.1 =>   /lib/64/libaio.so.1
        libm.so.2 =>     /lib/64/libm.so.2
        libthread.so.1 =>        /lib/64/libthread.so.1
        libmp.so.2 =>    /lib/64/libmp.so.2
        libmd.so.1 =>    /lib/64/libmd.so.1
        libscf.so.1 =>   /lib/64/libscf.so.1
        libdoor.so.1 =>  /lib/64/libdoor.so.1
        libuutil.so.1 =>         /lib/64/libuutil.so.1
        /platform/SUNW,Sun-Fire-V240/lib/sparcv9/libc_psr.so.1
        /platform/SUNW,Sun-Fire-V240/lib/sparcv9/libmd_psr.so.1

Set the LD_LIBRARY_PATH to resolve the issue.

bash
export ORACLE_HOME=/u01/prod/rdbms/10.2.0
export ORACLE_SID=appscdg
export LD_LIBRARY_PATH=/u01/prod/rdbms/10.2.0/lib:/usr/dt/lib:/usr/openwin/lib:/u01/prod/rdbms/10.2.0/ctx/lib
export PATH=$ORACLE_HOME/bin:$PATH

Now check the binary executable

bash-3.00$ ldd $ORACLE_HOME/bin/sqlplus
        libsqlplus.so =>
/oracle02/appslos/rdbms/10.2.0/lib/libsqlplus.so
        libclntsh.so.10.1 =>
/oracle02/appslos/rdbms/10.2.0/lib/libclntsh.so.10.1
        libnnz10.so =>   /oracle02/appslos/rdbms/10.2.0/lib/libnnz10.so
        libkstat.so.1 =>         /lib/64/libkstat.so.1
        libnsl.so.1 =>   /lib/64/libnsl.so.1
        libsocket.so.1 =>        /lib/64/libsocket.so.1
        libgen.so.1 =>   /lib/64/libgen.so.1
        libdl.so.1 =>    /lib/64/libdl.so.1
        libsched.so.1 =>         /usr/lib/64/libsched.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        libaio.so.1 =>   /lib/64/libaio.so.1
        libm.so.2 =>     /lib/64/libm.so.2
        libthread.so.1 =>        /lib/64/libthread.so.1
        libm.so.1 =>     /lib/64/libm.so.1
        librt.so.1 =>    /lib/64/librt.so.1
        libmp.so.2 =>    /lib/64/libmp.so.2
        libmd.so.1 =>    /lib/64/libmd.so.1
        libscf.so.1 =>   /lib/64/libscf.so.1
        libdoor.so.1 =>  /lib/64/libdoor.so.1
        libuutil.so.1 =>         /lib/64/libuutil.so.1
        /platform/SUNW,Sun-Fire-V240/lib/sparcv9/libc_psr.so.1
        /platform/SUNW,Sun-Fire-V240/lib/sparcv9/libmd_psr.so.1

Issue resolved.

libclntsh.so.10.1: wrong ELF class: ELFCLASS64 Killed

I was upgrading to Oracle Applications 11.5.10.2. to R12 (12.1.1) on Solaris 10 64-bit Operating System.

While invoking adadmin utility after sourcing the R12 apps environment, I faced the following error:

bash-3.00$ adadmin
ld.so.1: adadmin: fatal: /oracle02/appsdev/R12/apps/tech_st/10.1.2/lib/libclntsh.so.10.1: wrong ELF class: ELFCLASS64
Killed

Tried doing the below:

bash-3.00$ $ORACLE_HOME/bin/genclntsh -32
cat: cannot open /oracle02/appsdev/R12/apps/tech_st/10.1.3/lib32/ldflags
Undefined                       first referenced
 symbol                             in file
nnfhboot                            /oracle02/appsdev/R12/apps/tech_st/10.1.3/lib32/libn10.a(nnfgt.o)
nnflboot                            /oracle02/appsdev/R12/apps/tech_st/10.1.3/lib32/libn10.a(nnfgt.o)
nnflgapc                            /oracle02/appsdev/R12/apps/tech_st/10.1.3/lib32/libnl10.a(nlpa.o)
nnftboot                            /oracle02/appsdev/R12/apps/tech_st/10.1.3/lib32/libn10.a(nnfgt.o)
nnflgav                             /oracle02/appsdev/R12/apps/tech_st/10.1.3/lib32/libnl10.a(nlol.o)
nnflfrm                             /oracle02/appsdev/R12/apps/tech_st/10.1.3/lib32/libnl10.a(nlol.o)
nnflrne                             /oracle02/appsdev/R12/apps/tech_st/10.1.3/lib32/libnl10.a(nlol.o)
ntusini                             /oracle02/appsdev/R12/apps/tech_st/10.1.3/lib32/libn10.a(ntcontab.o)
ntpini                              /oracle02/appsdev/R12/apps/tech_st/10.1.3/lib32/libn10.a(ntcontab.o)
nttini                              /oracle02/appsdev/R12/apps/tech_st/10.1.3/lib32/libn10.a(ntcontab.o)
ntzini                              /oracle02/appsdev/R12/apps/tech_st/10.1.3/lib32/libn10.a(ntcontab.o)
ld: fatal: Symbol referencing errors. No output written to /oracle02/appsdev/R12/apps/tech_st/10.1.3/lib32/libclntsh.so.10.1

@Per ‘unpublished’ Bug 5665338 – This issue is due to the libclntsh.so.10.1 library not being present under the 10.1.2 ORACLE_HOME/lib32 folder.

To fix the issue,  I did the following:

1.  Created a symbolic link

bash-3.00$ rm $ORACLE_HOME/lib32/ldflags
bash-3.00$ ln -s $ORACLE_HOME/lib/ldflags $ORACLE_HOME/lib32/ldflags
bash-3.00$ ls -ltr $ORACLE_HOME/lib32/ldflags
lrwxrwxrwx   1 appsdev  oracle         53 Apr  4 09:17 /oracle02/appsdev/R12/apps/tech_st/10.1.2/lib32/ldflags -> /oracle02/appsdev/R12/apps/tech_st/10.1.2/lib/ldflags

2. As root user, created a soft link

bash-3.00$ which make
/usr/css/bin/make
bash-3.00$  ln -s /usr/css/bin/make /usr/bin/make

2.  Tested the fix

bash-3.00$ $ genclntsh

3.  The above genclntsh command completed without error, then relinked all of the software

bash-3.00$ relink all

Check if libclntsh.so.10.1 library has been created under the 10.1.2 ORACLE_HOME/lib32 folder.

bash-3.00$  ls -ltr $ORACLE_HOME/lib32/libclntsh.so.10.1
-rwxr-xr-x   1 appsdev  oracle    18681000 Apr  4 09:23 /oracle02/appsdev/R12/apps/tech_st/10.1.2/lib32/libclntsh.so.10.1

Invoked ad utlilities now:

bash-3.00$ adadmin

                     Copyright (c) 2002 Oracle Corporation
                        Redwood Shores, California, USA

                     Oracle Applications AD Administration

                                 Version 12.0.0

NOTE: You may not use this utility for custom development
      unless you have written permission from Oracle Corporation.

Your default directory is '/oracle02/appsdev/R12/apps/apps_st/appl'.
Is this the correct APPL_TOP [Yes] ?

The problem got fixed for me.

References: 413109.1, 395731.1