Skip to Main Content

Breadcrumb

Question and Answer

Connor McDonald

Thanks for the question, Mike.

Asked: December 01, 2016 - 9:14 pm UTC

Last updated: August 08, 2019 - 3:25 am UTC

Version: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

Viewed 1000+ times

You Asked

Hello,

I am fairly new to Oracle and quite frankly a noob at writing code. I am writing a query to pull in data. One of the fields needs to be concatenated. However there a multiple duplicate records in the field resulting in many rows of data. The original code that was given to me used a select distinct with a nested query to pull the data. I got around the select distinct by using a regexp_replace. In addition, I need to concatenate the rows into one field. If I run:

SELECT
regexp_replace(
listagg(
MASTARS_GROUP.GROUP_NAME, ',') within group (ORDER BY MASTARS_GROUP.GROUP_NAME)
,'([^,]+)(,\1)*(,|$)', '\1\3') as "Approval Group Needed"
FROM CONTACT, MASTARS_GROUP, GROUP_PERSON
WHERE 0=0
AND CONTACT.IS_APPROVER = 'Y'
AND CONTACT.IS_VISIBLE = 'Y'
AND CONTACT.STATUS = 'W'
AND CONTACT.REQUEST_ID = :RequestId
AND MASTARS_GROUP.GROUP_ID = CONTACT.GROUP_ID
AND CONTACT.PERSON_ID = GROUP_PERSON.PERSON_ID
AND CONTACT.GROUP_ID NOT IN (SELECT GROUP_ID FROM CONTACT WHERE STATUS != 'W' AND REQUEST_ID = :RequestId AND GROUP_ID IS NOT NULL);

the output is fine. However, this is just for one record. I need to run this in a larger query and when I incorporated it into my query, I am back to the dreaded ORA-01489: result of string concatenation is too long error.

Here is my complete query:

select L.site_code,R.request_id, e.event_id,
TO_CHAR(CAST((FROM_TZ(CAST( r.create_date AS TIMESTAMP),'+00:00') AT TIME ZONE 'US/Eastern') AS DATE),'mm/dd/yyyy HH24:MI:SS') as "Create US/Eastern",
TO_CHAR(CAST((FROM_TZ(CAST(e.schedule_start AS TIMESTAMP),'+00:00') AT TIME ZONE 'US/Eastern') AS DATE),'mm/dd/yyyy HH24:MI:SS') as "Schedule Start US/Eastern",
TO_CHAR(CAST((FROM_TZ(CAST(e.schedule_stop AS TIMESTAMP),'+00:00') AT TIME ZONE 'US/Eastern') AS DATE),'mm/dd/yyyy HH24:MI:SS') as "Schedule Stop US/Eastern",
e.schedule_start as "Schedule Start GMT", e.schedule_stop as "Schedule Stop GMT", r.approval_status,r.status, c.name as CompleteStatus,
(CASE when e.schedule_type = 'N' then 'Nightly'
when e.schedule_type = 'D' then 'Demand'
ELSE 'Other' END ) as "Schedule Type",
ec.description, dm.impact_type,n.text as work_desc,
(regexp_replace(listagg( mg.GROUP_NAME, ',') within group (ORDER BY mg.GROUP_NAME) ,'([^,]+)(,\1)*(,|$)', '\1\3')) as AprovalNeeded
from request R , event e, network n, location L, event_log eL, Completion_code c, event_code ec, item i, digital_media_risk_default dm, request_log rl, note_req n,contact ctc,mastars_group mg,group_person gp
where
R.request_id = e.request_id
and e.event_id = L.event_id
and r.request_id = i.request_id
and rl.request_id = e.request_id
and rl.work_desc = n.note_id
and n.network_id = r.network_id
AND e.EVENT_ID = eL.EVENT_ID
and e.event_code= dm.event_code(+)
and e.event_code = ec.event_code
AND eL.COMP_STATUS = c.COMPLETION_CODE_ID
and ctc.IS_APPROVER = 'Y'
AND ctc.IS_VISIBLE = 'Y'
AND ctc.STATUS = 'W'
AND ctc.REQUEST_ID = r.request_id
AND mg.GROUP_ID = ctc.GROUP_ID
AND ctc.PERSON_ID = gp.PERSON_ID
and n.network = 'DIGITAL MEDIA'
and r.approval_status in ('A','W')
and r.status not in ('C')
and FROM_TZ(CAST(e.schedule_start AS TIMESTAMP),'UTC') AT TIME ZONE 'US/Eastern' >= to_date(:StartRange,'mm/dd/yyyy hh24:mi:ss')
and FROM_TZ(CAST(e.schedule_start AS TIMESTAMP),'UTC') AT TIME ZONE 'US/Eastern' <= to_date(:StopRange,'mm/dd/yyyy hh24:mi ss')
and n.network_id not in (3351,2950)
group by L.site_code , r.request_id, e.event_id,r.create_date, e.schedule_start, e.schedule_stop, r.approval_status, r.status,c.name ,e.schedule_type,ec.description, dm.impact_type, n.text
order by e.schedule_start ,r.status desc

I am back to square one with the dreaded error. I see you have talked about custom defined functions and I wonder if that my do the trick.

thanks

and Connor said...

Here's a link to create your own listagg that will return a clob.

http://sql-plsql-de.blogspot.com.au/2014/01/sql-listagg-mit-clob-ausgabe-kein.html

It's in German but the code is in English :-)

In terms of keeping the syntax simple, I'd use a WITH, so your query becomes something like:

WITH
  my_query as 
  ( select ...
    from   ...
  )
select col, listagg_clob()
from   my_query
group by col


So the grouping/concatention is separate from the extraction of the data.

(This is purely from a readability perpsective)

Rating

  (6 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

Easier methods

Stew Ashton, December 02, 2016 - 10:07 am UTC

Hello,

Mike has version 12c, which means both XMLDB and MATCH_RECOGNIZE are necessarily available.

When a list is too long for a VARCHAR2, I see three options:
1) Truncate at 4000 bytes max
2) Use a CLOB
3) Break the list into multiple VARCHAR2 pieces

For option 2, I suggest XMLAGG:
select object_type,
SUBSTR(
  xmlcast(
    xmlagg(
      xmlelement(E, ',' || owner || '.' || object_name)
      order by owner, object_name
    ) AS CLOB
  ), 2
) objects
from all_objects
group by object_type
order by object_type;

For option 3, I suggest MATCH_RECOGNIZE:
select object_type, piece,
  listagg(obj, ',') within group(order by obj) objects
from (
  select object_type, owner || '.' || object_name obj
  from all_objects
)
match_recognize(
  partition by object_type order by obj
  measures match_number() piece
  all rows per match
  pattern(a+)
  define a as sum(length(obj)) + count(*) - 1 <= 4000
)
group by object_type, piece
order by object_type, piece;

Best regards, Stew
Chris Saxon
December 02, 2016 - 5:03 pm UTC

Thanks for stopping by Stew.

Pre-12c you can also use the model clause for splitting the string up as explained at:

https://blogs.oracle.com/datawarehousing/entry/managing_overflows_in_listagg

LISTAGG and MATCH_RECOGNIZE

A reader, December 02, 2016 - 7:31 pm UTC

Can you please point me to an article which explains MATCH_RECOGNIZE in detail on how it works etc...

Also, for the LISTAGG example above(both with MATCH_RECOGNIZE and XMLAGG), it does not eliminate duplicates.

Eliminate duplicates with MATCH_RECOGNIZE

Stew Ashton, December 04, 2016 - 1:56 pm UTC

select object_type, piece,
  listagg(obj, ',') within group(order by obj) objects
from (
  select object_type, owner || '.' || object_name obj
  from all_objects
  where object_type = 'CLUSTER'
  union all
  select object_type, owner || '.' || object_name obj
  from all_objects
  where object_type = 'CLUSTER'
)
match_recognize(
  partition by object_type order by obj
  measures match_number() piece
  all rows per match
  pattern( ( {-a-} | b )+ )
  define a as obj = next(obj),
    b as sum(length(b.obj)) + count(b.*) - 1 <= 80 
)
group by object_type, piece
order by object_type, piece;

OBJECT  PIECE OBJECTS
CLUSTER 1     SYS.C_COBJ#,SYS.C_FILE#_BLOCK#,SYS.C_MLOG#,SYS.C_OBJ#,SYS.C_OBJ#_INTCOL#
CLUSTER 2     SYS.C_RG#,SYS.C_TOID_VERSION#,SYS.C_TS#,SYS.C_USER#,SYS.SMON_SCN_TO_TIME_AUX

link unavailable

Dan T., August 07, 2019 - 6:22 pm UTC

The link in the original reply is not available due to "This blog is open to invited readers only". Is there another source?

Also, the XMLAgg approach modifies the data to escape for HTML tags (i.e. '"' becomes '&quot;'). To unescape, it limits again to 4000 characters which defeats the original purpose.
Connor McDonald
August 08, 2019 - 3:25 am UTC

OK, try this one

SQL> create or replace type listagg_clob_t as object
  2  ( t_varchar2 varchar2(32767)
  3  , t_clob clob
  4  , static function odciaggregateinitialize( sctx in out listagg_clob_t )
  5    return number
  6  , member function odciaggregateiterate
  7      ( self in out listagg_clob_t
  8      , a_val varchar2
  9      )
 10    return number
 11  , member function odciaggregateterminate
 12      ( self in out listagg_clob_t
 13      , returnvalue out clob
 14      , flags in number
 15      )
 16    return number
 17  , member function odciaggregatemerge
 18      ( self in out listagg_clob_t
 19      , ctx2 in out listagg_clob_t
 20      )
 21    return number
 22  )
 23  /

Type created.

SQL>
SQL> create or replace type body listagg_clob_t
  2  is
  3    static function odciaggregateinitialize( sctx in out listagg_clob_t )
  4    return number
  5    is
  6    begin
  7      sctx := listagg_clob_t( null, null );
  8      return odciconst.success;
  9    end;
 10  --
 11    member function odciaggregateiterate
 12      ( self in out listagg_clob_t
 13      , a_val varchar2
 14      )
 15    return number
 16    is
 17      procedure add_val( p_val varchar2 )
 18      is
 19      begin
 20        if nvl( lengthb( self.t_varchar2 ), 0 ) + lengthb( p_val ) <= 4000
 21  -- Strange limit, the max size of self.t_varchar2 is 29993
 22  -- If you exceeds this number you get ORA-22813: operand value exceeds system limits
 23  -- with 29993 you get JSON-output as large 58894 bytes
 24  -- with 4000 you get JSON-output as large 1063896 bytes, probably max more
 25        then
 26          if self.t_varchar2 is null then
 27            self.t_varchar2 := self.t_varchar2 || p_val;
 28          else
 29            self.t_varchar2 := self.t_varchar2 || ',' || p_val;
 30          end if;
 31        else
 32          if self.t_clob is null
 33          then
 34            dbms_lob.createtemporary( self.t_clob, true, dbms_lob.call );
 35            dbms_lob.writeappend( self.t_clob, length( self.t_varchar2 ), self.t_varchar2 );
 36          else
 37            dbms_lob.writeappend( self.t_clob, length( self.t_varchar2 +1), ','||self.t_varchar2 );
 38          end if;
 39          self.t_varchar2 := p_val;
 40        end if;
 41      end;
 42    begin
 43      add_val( a_val );
 44      return odciconst.success;
 45    end;
 46  --
 47    member function odciaggregateterminate
 48      ( self in out listagg_clob_t
 49      , returnvalue out clob
 50      , flags in number
 51      )
 52    return number
 53    is
 54    begin
 55      if self.t_clob is null
 56      then
 57        dbms_lob.createtemporary( self.t_clob, true, dbms_lob.call );
 58      end if;
 59      if self.t_varchar2 is not null
 60      then
 61        dbms_lob.writeappend( self.t_clob, length( self.t_varchar2 ), self.t_varchar2 );
 62      end if;
 63      returnvalue := self.t_clob;
 64      return odciconst.success;
 65    end;
 66  --
 67    member function odciaggregatemerge
 68      ( self in out listagg_clob_t
 69      , ctx2 in out listagg_clob_t
 70      )
 71    return number
 72    is
 73    begin
 74      if self.t_clob is null
 75      then
 76        dbms_lob.createtemporary( self.t_clob, true, dbms_lob.call );
 77      end if;
 78      if self.t_varchar2 is not null
 79      then
 80        dbms_lob.writeappend( self.t_clob, length( self.t_varchar2 ), self.t_varchar2 );
 81      end if;
 82      if ctx2.t_clob is not null
 83      then
 84        dbms_lob.append( self.t_clob, ctx2.t_clob );
 85        dbms_lob.freetemporary( ctx2.t_clob );
 86      end if;
 87      if ctx2.t_varchar2 is not null
 88      then
 89        dbms_lob.writeappend( self.t_clob, length( ctx2.t_varchar2 ), ctx2.t_varchar2 );
 90        ctx2.t_varchar2 := null;
 91      end if;
 92      return odciconst.success;
 93    end;
 94  --
 95  end;
 96  /

Type body created.

SQL> sho err
No errors.
SQL>
SQL> create or replace function listagg_clob( agg varchar2 )
  2  return clob
  3  parallel_enable aggregate using listagg_clob_t;
  4  /

Function created.

SQL>
SQL> select listagg_clob(owner)
  2  from dba_objects
  3  where rownum <= 10;

LISTAGG_CLOB(OWNER)
--------------------------------------------------------------------------------
SYS,SYS,SYS,SYS,SYS,SYS,SYS,SYS,SYS,SYS

1 row selected.

SQL>
SQL>


Correcting the XMLAGG answer

Stew Ashton, August 10, 2019 - 8:33 am UTC

Thanks to Dan for pointing out the problem with the XMLAGG solution! This correction seems to work:
select object_type,
DBMS_XMLGEN.CONVERT(
  substr(
    xmlcast(
      xmlagg(
        xmlelement(E, ',' ||
          DBMS_XMLGEN.CONVERT(owner || '.' || object_name || '"#$%&', 0))
        order by owner, object_name
      ) AS CLOB
    ), 2
  )
  , 1
) objects
from all_objects
group by object_type
order by object_type;

OBJECT_TYPE  OBJECTS
CLUSTER      SYS.C_COBJ#"#$%&,SYS.C_FILE#_BLOCK#"#$%&,...

Best regards,
Stew Ashton

And using JSON

Stew Ashton, August 10, 2019 - 2:11 pm UTC

This should work starting with version 12.2 patched. It is at least three times faster than the XML solution.
select object_type,
json_value(
  replace(
    json_arrayagg(owner || '.' || object_name || '"' returning clob),
    '","',
    ','
  ),
  '$[0]' returning clob
) objects
from all_objects
group by object_type
order by object_type;
Best regards,
Stew

More to Explore

CLOB/BLOB/etc

Complete documentation on Securefiles and Large Objects here