Troubleshooting ORA-12547 TNS: Lost Contact (Doc ID 555565.1) To BottomTo Bottom 

In this Document
Purpose
Troubleshooting Steps
  Bequeath (Local) :
  Remote Connections:
  Listener specific : 
  Permissions:
  Diagnosis :
References
APPLIES TO:

Oracle Net Services - Version 10.2.0.5 to 12.1.0.2 [Release 10.2 to 12.1]
Information in this document applies to any platform.
PURPOSE

Step by Step Troubleshooting Guide for ORA-12547 TNS LOST CONTACT error faced while using Oracle Database server.

TROUBLESHOOTING STEPS

Background / Why ORA-12547 

12547 indicates that the communication channel has been broken. 
It's most often thrown because the other end of the process went away unexpectedly. 


Known Scenarios / Where ORA-12547 can Occur 
This error can occur in following scenarios:

Bequeath (Local) :

The following/below are common scenarios in which 12547 occurs while making a BEQ 
connection. 

Problem:- 
BEQ connection fails when connecting with / as sysdba 

Cause:- 
Oracle binaries have not been linked incorrectly 

Solution :- 
relink the Oracle binaries by executing the following command : 

$ ORACLE_HOME/bin relink all 

(OR) 

Cause :- 
Shared memory segment and/or semaphores are not flushed properly. 

A system level trace taken while connecting to the database " / as sysdba " reveals the following :

2654: write(5, " O R A - 0 0 6 0 0 : i".., 81) = 81 
2654: write(5, "\n", 1) = 1 
2654: write(5, " O R A - 2 7 1 0 0 : s".., 45) = 45
Solution :- 
Please identify and clear the problematic memory segment and semaphores. 

The following metalink document describes the steps in detail. 
Note 381566.1 connect / as sysdba Fails with Ora-12547 And Tns-12514 

(OR)

Cause :-
External Shared Memory is configured in the environment.Extended Shared Memory is not supported by Oracle and should not be set/exist in the environment.

Truss of sqlplus connection shows a trace file being generated in $ORACLE_HOME/rdbms/log directory instead of the BDUMP location.

Excerpt from truss output:

569536: statx("/opt/oracle/oracle/product/11.1.0/rdbms/log/mdsp_ora_569536.trc", 0x0FFFFFFFFFFF7DF0, 176, 01) = 0
569536: statx("/opt/oracle/oracle/product/11.1.0/rdbms/log/mdsp_ora_569536.trc", 0x0FFFFFFFFFFF7DF0, 176, 0) = 0
569536: open("/opt/oracle/oracle/product/11.1.0/rdbms/log/mdsp_ora_569536.trc", O_WRONLY|O_APPEND|O_LARGEFILE) = 3
569536: kwrite(3, " D u m p f i l e ", 10) = 10
569536: kwrite(3, " / o p t / o r a c l e /".., 63) = 63
569536: kwrite(3, "\n", 1) = 1
569536: kwrite(3, "\n * * * 2 0 0 9 - 0 2".., 29) = 29
569536: kwrite(3, " O r a c l e D a t a b".., 200) = 200
Solution :-
1. Unset the EXTSHM environment variable in the session where you are connecting to database:
unsetenv EXTSHM
2. Run sqlplus:
sqlplus / as sysdba

(OR)

Cause :-
The ulimit settings for 'open files' is inadequate:

Truss on sqlplus may look like :


4074: shmat(9, 0x380000000, 040000) = 0x380000000
4074: Incurred fault #6, FLTBOUNDS %pc = 0x100E59824
4074: siginfo: SIGSEGV SEGV_MAPERR addr=0x000006C0
4074: Received signal #11, SIGSEGV [caught]
4074: siginfo: SIGSEGV SEGV_MAPERR addr=0x000006C0
4074: lwp_sigmask(SIG_SETMASK, 0x9FBEF457, 0x0000FFF7) = 0xFFBFFEFF [0x0000FFFF]
4074: lwp_sigmask(SIG_SETMASK, 0x9FBEF057, 0x0000FFF7) = 0xFFBFFEFF [0x0000FFFF]
4074: Incurred fault #6, FLTBOUNDS %pc = 0x100E5BD3C
4074: siginfo: SIGSEGV SEGV_MAPERR addr=0x00000008 
 Solution:-

Increase the open files resource limit to a higher value with ulimit

(OR)


Cause :-
See the following issue:

Bug 4335746 STARTUP GIVES TNS LOST CONTACT , TRACES GIVE SKGM ERROR 27148 WITH ERROR CODE 1

Collect a truss of the failure using:

% truss -aefo /tmp/sqlplus.out sqlplus / as sysdba
 

sqlplus.out 
----------- 
23: lxstat(2, "/rdbms1/ora10gr2/rdbms/log/testdb_ora_23.trc", 0x08045AB8) Err#2 ENOENT 
23: xstat(2, "/rdbms1/ora10gr2/rdbms/log/testdb_ora_23.trc", 0x08045AB8) Err#2 ENOENT 
23: open("/rdbms1/ora10gr2/rdbms/log/testdb_ora_23.trc", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 
0660) = 3 
23: write(3, 0x0BED1F20, 0) = 0 
23: write(3, " / r d b m s 1 / o r a 1".., 44) = 44 
23: write(3, "\n", 1) = 1 
23: write(3, " O r a c l e D a t a b".., 122) = 122 
23: write(3, "\n", 1) = 1 
23: write(3, " O R A C L E _ H O M E ".., 31) = 31 
23: uname(0x0C4BF314) = 1 
23: write(3, " S y s t e m n a m e :".., 19) = 19 
23: write(3, " N o d e n a m e :\t o".., 19) = 19 
23: write(3, " R e l e a s e :\t 5 . 1".., 14) = 14 
23: write(3, " V e r s i o n :\t G e n".., 27) = 27 
23: write(3, " M a c h i n e :\t i 8 6".., 15) = 15 
23: write(3, " I n s t a n c e n a m".., 22) = 22 
23: write(3, " R e d o t h r e a d ".., 47) = 47 
23: write(3, " O r a c l e p r o c e".., 25) = 25 
23: write(3, " U n i x p r o c e s s".., 43) = 43 
23: write(3, "\n", 1) = 1 
23: write(3, "\n", 1) = 1 
23: brk(0x0C4DC778) = 0 
23: brk(0x0C4DE778) = 0 
23: open("/usr/share/lib/zoneinfo/US/Pacific", O_RDONLY) = 5 
23: fstat64(5, 0x08045BD0) = 0 
23: read(5, " T Z i f\0\0\0\0\0\0\0\0".., 1017) = 1017 
23: close(5) = 0 
23: write(3, " * * * 2 0 0 8 - 1 0 -".., 27) = 27 
23: write(3, "\n", 1) = 1 
23: write(3, " s k g m e r r o r 2".., 46) = 46 
23: write(3, "\n", 1) = 1 
23: _exit(0) 
20: read(11, 0x0803FAF0, 64) = 0 
20: close(11) = 0 
20: close(10) = 0 
20: getpid() = 20 [29981]

 

 Solution:-
Confirm what the current ulimit setting is for stack


% ulimit -a
Cause :-  /etc/fstab is mounted with the nosuid option.  All local connections fail with ORA-12547

Solution:-
Replacing it with suid fixed the issue.

 


Note 188149.1 How to Display and Change UNIX Process Resource Limits

Check the install guide for your specific platform and version of Oracle and set stack as stated.

     example:  ulimit -s -1 ** This is Unlimited, the recommended setting on AIX **

Note 188149.1 How to Display and Change UNIX Process Resource Limits

 

Remote Connections:

The following/below are common scenarios in which 12547 occurs while making a TCP connection. 

Problem :- 
Remote connections to the database server fail with ORA-12547 

Cause :- 
SQLNET.INBOUND_CONNECT_TIMEOUT and/or INBOUND_CONNECT_TIMEOUT_listener_name is set in the database server's sqlnet.ora and listener.ora 

If the client fails to establish a connection and complete authentication in the time specified, then the database server terminates the connection. 

In 10g and higher, you may see ORA-609 or ORA-3136 errors in the alert.log.

Solution :- 
Tune the parameters SQLNET.INBOUND_CONNECT_TIMEOUT and/or INBOUND_CONNECT_TIMEOUT_listener_name to appropriate values.

(OR)

Cause :- 
TCP.VALIDNODE_CHECKING is active on the database server and the TCP.INVITED_NODES list does not have the IP address of the failing client. 
Alternatively the TCP.EXCLUDED_NODES list contain the IP address of the failing client. 

Solution :- 
Either add the IP address of the failing client in the TCP.INVITED_NODES list or remove it from the TCP.EXCLUDED_NODES list

It is recommended to restart (not reload)  the database listener for these parameters to take effect.

Listener specific : 

Problem :- 
Listener is failing to start with the error ORA-12547 and  Error: 104: Connection reset by peer  

Cause :- 
In the strace trace file of the tnslsnr executable, we see segmentation fault after following calls:


28567      0.000054 open("/u01/home/oracle/ldaprc", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000021>
28567      0.000068 open("/u01/home/oracle/.ldaprc", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000020>
28567      0.000063 open("ldaprc", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000018>
28567      0.000084 --- SIGSEGV (Segmentation fault) @ 0 (0) ---

And we also see the first reference to ldap files being made after reading file nsswitch.conf


Solution :- 

LDAP is not used for authentication and not configured. However, in nsswitch.conf file we have some entries with ldap resolution. The solution is to modify nsswitch.conf to assure there is no reference to ldap.


Problem :- 
Listener is failing to start with the error ORA-12547. 

Cause :- 
TCP.VALIDNODE_CHECKING is active on the database server and the TCP.INVITED_NODES list does not have the IP address of the Database server specified in the address definition of the listener. 
Alternatively the TCP.EXCLUDED_NODES list contain the IP address of the Database server specified in the address definition of the listener. 

Solution :- 
Either add the IP address of the Database server in the TCP.INVITED_NODES list and remove it from the TCP.EXCLUDED_NODES list if present 

The listener must be restarted for these changes to take effect. 

Problem :- 
Tnsping to the database listener is failing with ORA-12547

TNS Ping Utility for Linux: Version 10.1.0.5.0 - Production on 11-AUG-2016 04:01:33

Used parameter files:

/u01/oracle/product/10.1.0.5/db_1/network/admin/sqlnet.ora

Used TNSNAMES adapter to resolve the alias

Attempting to contact (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=X.X.X.X)(PORT=10710)) (CONNECT_DATA= (SERVICE_NAME=myservice.oracle.com) (INSTANCE_NAME=myservice)))

TNS-12547: TNS:lost contact

Cause :-
TCP.VALIDNODE_CHECKING is active on the database server and the TCP.INVITED_NODES list does not have the IP address of the client that is attempting this tnsping.  Alternatively, TCP.EXCLUDED_NODES
list contains this client's ip address.

Solution :- 
Add the client's ip address to the server side sqlnet.ora file in the TCP.INVITED_NODES list OR remove it from the TCP.EXCLUDED_NODES list if present 
The listener must be restarted for these changes to take effect.  

Problem:-

After changing variables in /etc/system and rebooting), a 10g or newer Listener will not start. 


lsnrctl> start 
Starting /u01/oracle/product/10.2.0/Db_1/bin/tnslsnr: please wait... 
TNSLSNR for Solaris: Version 10.2.0.1.0 - Production 
System parameter file is /u01/oracle/product/10.2.0/Db_1/network/admin/listener.ora 
Log messages written to /u01/oracle/product/10.2.0/Db_1/network/log/listener.log 
Trace information written to /u01/oracle/product/10.2.0/Db_1/network/trace/listener.trc 
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=server11)(PORT=1521))) 
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=extproc10))) 

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=server11)(PORT=1521))) 
TNS-12547: TNS:lost contact 
TNS-12560: TNS:protocol adapter error 
TNS-00517: Lost contact 
Solaris Error: 131: Connection reset by peer 
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=extproc10))) 
TNS-12541: TNS:no listener 
TNS-12560: TNS:protocol adapter error 
TNS-00511: No listener 
Solaris Error: 146: Connection refused

Cause:-
Issue is due to a DNS client process, 'BIND 9' not running/started on the Solaris 10 Server. 

"BIND allows DNS clients and applications to query DNS servers for the IPv4 and IPv6 networks. 
BIND includes two main components: a stub resolver API, resolver(3resolv), and the DNS name server 
with various DNS tools." 

Reference: 
http://docs.sun.com/app/docs/doc/817-0547/6mgbdbsna?a=view 
Solaris 10 Release and Installation Collection --> Solaris 10 What's New --> 2. What's New in the 
Solaris 10 3/05 Release -->Freeware Enhancements 

Solution:-
Contact the System Administrator to start the BIND 9 DNS Client on the Solaris10 Server.

Permissions:

Problem:-  
Connections via the listener are failing with ORA-12547.  It is likely in this scenario that LOCAL or BEQ connections to the instance are successful.

Cause:-
The permissions on the Oracle binary and not correct and the listener cannot spawn a server process.  This is common in environments where the listener is running in the GRID home
and servicing connections to an instance in a different $ORACLE_HOME.  
See the following document for both RAC/SCAN:  
Note 1069517.1 ORA-12537 if Listener (including SCAN Listener) and Database are Owned by Different OS User

Issue the following checks:

$ ls -l $RDBMS_HOME/bin/oracle
ls: /home/oracle/app/oracle/product/11.2/db/bin/oracle: Permission denied

1. Listener owner (including SCAN listener) cannot access oracle binary in database home:

As listener owner:

$ ls -l $RDBMS_HOME/bin/oracle
ls: /home/oracle/app/oracle/product/11.2/db/bin/oracle: Permission denied


2. Oracle binary in database home has wrong permission:

ls -l $RDBMS_HOME/bin/oracle
-rwxr-x--x 1 oracle asmadmin 184286251 Aug  9 16:25 /home/oracle/app/oracle/product/11.2/db/bin/oracle
The permission "-rwxr-x--x" is wrong as it's missing suid bit, oracle binary should have permission of 6751:

-rwsr-s--x 1 oracle asmadmin 184286251 Aug  9 16:25 /home/oracle/app/oracle/product/11.2/db/bin/oracle

Solution:-

 Solution is to make sure file system for database home has setuid/suid set, database binary($RDBMS_HOME/bin/oracle) has correct ownership and permission, and listener owner is able to access database oracle binary (as listener owner, "ls -l $RDBMS_HOME/bin/oracle" will tell)


Diagnosis :


If the above discussed scenarios did not help than please collect the following diagnostic information and the provide the same while logging a service request . 

The diagnosis on this error depends on the context or the scenario in which the error is occurring: 

1. If the error is occurring during the connection establishment from a remote installation than 

a. Enable a support level client, listener and server side SQLNET traces. 
Reproduce the issue or wait until the error reoccurs and upload the corresponding traces . 
Note 395525.1 How to Enable Oracle SQLNet Client , Server , Listener , Kerberos and External procedure Tracing from Net Manager 

b. Upload the SQLNET.ORA, LISTENER.ORA of the database server. 

c. From the client where connection fails, check whether telnet to the listener port works.(expected to bring a blank screen) 
$ telnet <database server hostname/ IP> <listener port> 


2. If the error is occurring on a remote installation after connection establishment than 

a. Enable a support level client and server side SQLNET traces. 
Reproduce the issue or wait until the error reoccurs and upload the corresponding traces . 

b. From the client where connection fails, check whether telnet or putty to the database server also fails with idle time. 
$ telnet <database server hostname/ IP> 


3. If the error occurs while performing listener operations than 

a. Enable a support level Listener traces. 
Reproduce the issue and upload the corresponding traces.