Hi TOM,

that is the error that we get:

SQL:
UPDATE /*+ PARALLEL */ POM_BACKPOINTER SET to_class = 1546 WHERE to_uid IN ( SELECT puid FROM ao_temp );

ERROR:
ORA-00600: internal error code, arguments: [kdu_array_buf:depth], [17], [16], [], [], [], [], [], [], [], [], []
(for one of the parallel slaves)

qerpxSlaveFetch
rwsrid:1 pxid:1 err:600
----- Explain Plan Dump -----
----- Compact Format (Stream) -----

We took a dump of the production database and imported it into a dryrun system. The same dump does not show the problem in Test or Dev System. So we assume it is a Problem if this Database Instance.

We tried to reduce the dop, we increased the search area, recreate the statistics and the indexes, changed the execution plan... no change.

Also manual execution of the same SQLs did not show the same issue.

I was thinking that it is maybe realted to the UNDO Tablespace, but I don't know how to proof it or what to do now. Should we try to recreate the UNDO Tablespace? Any other Idea for the root cause?

Some snippets of the trace that I thought could be interesting:

Problem Key: ORA 600 [kdu_array_buf:depth]
Error: ORA-600 [kdu_array_buf:depth] [17] [16] [] [] [] [] [] [] [] [] []
[00]: dbgexExplicitEndInc [diag_dde]
[01]: dbgeEndDDEInvocationImpl [diag_dde]
[02]: dbgeEndDDEInvocation [diag_dde]
[03]: kdu_array_buf [Data]<-- Signaling
[04]: kduovw [Data]
[05]: kduurp [Data]
[06]: kdusru [Data]
[07]: kdu_retry_update [Data]
[08]: kdu_array_flush_retry [Data]
[09]: kdu_retry_update [Data]
[10]: kdu_array_flush_retry [Data]
[11]: kdu_retry_update [Data]
[12]: kdu_array_flush_retry [Data]
[13]: kdu_retry_update [Data]
[14]: kdu_array_flush_retry [Data]
[15]: kdu_retry_update [Data]
[16]: kdu_array_flush_retry [Data]
[17]: kdu_retry_update [Data]
[18]: kdu_array_flush_retry [Data]
[19]: kdu_retry_update [Data]
[20]: kdu_array_flush_retry [Data]
[21]: kdu_retry_update [Data]
[22]: kdu_array_flush_retry [Data]
[23]: kdu_retry_update [Data]
[24]: kdu_array_flush_retry [Data]
[25]: kdu_retry_update [Data]
[26]: kdu_array_flush_retry [Data]
[27]: kdu_retry_update [Data]
[28]: kdu_array_flush_retry [Data]
[29]: kdu_retry_update [Data]
[30]: kdu_array_flush_retry [Data]
[31]: kdu_retry_update [Data]
[32]: kdu_array_flush_retry [Data]
[33]: kdu_retry_update [Data]
[34]: kdu_array_flush_retry [Data]
[35]: kdu_retry_update [Data]
[36]: kdu_array_flush_retry [Data]
[37]: kdu_retry_update [Data]
[38]: kdu_array_flush_retry [Data]
[39]: kdu_retry_update [Data]
[40]: kdu_array_flush_retry [Data]
[41]: kdu_array_flush1 [Data]
[42]: kdusru [Data]
[43]: kauupd [Data]
[44]: updrow [DML]
[45]: qerupFetch [SQL_Execution]
[46]: qertqoFetch [Parallel_Execution]
[47]: qerpxSlaveFetch [Parallel_Execution]
[48]: qerpxFetch [Parallel_Execution]
[49]: updexe [DML]
--

kdu_array_buf: recursion depth failure
************************************************************
ORA-30036 DIAGNOSTIC
This diagnostic information is dumped to trace file at
most once every 24 hours, it does not indicate any error.
************************************************************
Reason: kdu_array_buf: recursion depth failure
Current undo tablespace UNDOTBS1 (tsn=2)
undo tablespace current size 9830400 blks, maxsize 12582906 blks, extensiable
Undo Retention (reactive):2723, Max Query Length:2684
Parameter Undo Retention:900, Tuned Undo Retention:2723, High threshold Undo Retention:4294967294 autotune:1
Retention Guarantee FALSE
Current Time is 1518190661

SYMPTOMS
The following error is reported by alert log

ORA-00600: internal error code, arguments: [kdu_array_buf:depth], [17], [16], [], [], [], [], [], [], [], [], []

The trace file confirms:

Call Stack Trace
----------------
kdu_array_buf kduurp kdusru kdu_retry_update kdu_array_flush ...


----- Incident Context Dump -----
...
[01]: kdu_array_buf []<-- Signaling
[02]: kduurp []
[03]: kdusru []
[04]: kdu_retry_update []
[05]: kdu_array_flush []
[06]: kdu_retry_update []
[07]: kdu_array_flush []
[08]: kdu_retry_update []
[09]: kdu_array_flush []
[10]: kdu_retry_update []
...

CHANGES
The ORA-600 occurs when SMON performs maintenance on Undo Tablespace (such as shrinking an Undo Segment). This might not be obvious when reading the trace file as this might show a DML statement such as illustrated by the following trace information:

----- Current SQL Statement for this session (sql_id=1vwg99k24trd0) -----
update "OPI"."MLOG$_OPI_DBI_JOB_MTL_DTL_" set snaptime$$ = :1 where rowid in 
(select rowid from "OPI"."MLOG$_OPI_DBI_JOB_MTL_DTL_" AS OF SNAPSHOT (:2)
log$ where snaptime$$ >
to_date('2100-01-01:00:00:00','YYYY-MM-DD:HH24:MI:SS'))
----- PL/SQL Stack -----
----- PL/SQL Call Stack -----
object line object
handle number name
47e5d9948 84 package body SYS.DBMS_SNAPSHOT
47e5d9948 1808 package body SYS.DBMS_SNAPSHOT
47e5d9948 2506 package body SYS.DBMS_SNAPSHOT
47e5d9948 2751 package body SYS.DBMS_SNAPSHOT
47e5d9948 2720 package body SYS.DBMS_SNAPSHOT
...

CAUSE
This issue is addressed by Bug:10410505 which is a confirmed duplicate of unpublished Bug 8476681 which was still not fixed at the time of publishing this article.

WORKAROUND
This fix requires architectural changes which is not possible for 10g/11g. Therefore the following workaround could be used: 

Set parameter "_kdu_array_depth" to a value larger then the default value (16). 
The parameter is dynamic and can be adjusted without a database restart.

Example: Set "_kdu_array_depth" to 24 

alter system set "_kdu_array_depth"=24 scope=memory;

To implement this into spfile, execute:

alter system set "_kdu_array_depth"=24 scope=spfile;