We have a Java application that uses a connection pool with Oracle database, during the week these conections are frequently refreshed (closed and opened again), the problem is at some point (after two or three days) the application is unable to acquire new connections, it get stuck on that line:
Runner004" prio=10 tid=0x00007f4b2c119800 nid=0x12b6 runnable [0x00007f4b00151000]
java.lang.Thread.State: RUNNABLE
at oracle.jdbc.driver.T2CConnection.t2cCreateState(Native Method)
at oracle.jdbc.driver.T2CConnection.logon(T2CConnection.java:530)
at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:662)
at oracle.jdbc.driver.T2CDriverExtension.getConnection(T2CDriverExtension.java:54)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:560)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at TesteDB.getConnection(TesteDB.java:238)
at TesteDB$Runner.run(TesteDB.java:285)Version information:* Oracle Instant Client 12.1.0.2.0 on Linux.
* The database version is 11.2.0 or 10.2.0.
Troubleshooting we did, when the database stopped to respond:* Connect through SQLPLUS doesn't respond anything. It just waits forever.
* TNSPING doesn't respond anything.
* lsnrctl status doesn't respond anything.
* Nothing on alert.log
* We rebooted the listener, same problem.
* We rebooted the database service, it started to work.
We have absolutely no idea what we should look into, any suggestions?
If your database is *totally* stuck, then you might still be able to do a hanganalyze.
Tanel has a nice write up here on that
http://tech.e2sn.com/oracle/troubleshooting/hang/how-to-log-on-even-when-sysdba-can-t-do-so Then log a Support call and provide them that dump.
But if even lsnrctl doesn't respond, that sounds more like an OS level issue (since lsnrctl does not talk to the database). So you might be into a OS troubleshooting exercise there with something like strace or truss.
Hope this helps.