Skip to Main Content

Breadcrumb

Question and Answer

Connor McDonald

Thanks for the question, Vipin.

Asked: October 07, 2016 - 3:03 pm UTC

Last updated: November 10, 2021 - 4:12 am UTC

Version: 12C

Viewed 10K+ times! This question is

You Asked

Hi Tom,
This is my first question to you and hope to receive a positive response :)

Straight to the Question:

I have a table named trail_log. This is a huge table and a daily growth of this table is 12GB Approx. Currently 76216160 rows. Important here to note is that this is online table in which 24*7 insert statement are coming from front end application. ONLY INSERTS come from front end application.

Now we have a archival logic where we need to delete data from this table older than 15 days: A manual JOB

Table Structure

CREATE TABLE TRAIL_LOG
(
ORDER_NO NUMBER(9),
BS_NAME VARCHAR2(50 BYTE),
MESSAGE_SENDER VARCHAR2(25 BYTE),
MESSAGE_RECEIVER VARCHAR2(25 BYTE),
TRANSACTION_ID VARCHAR2(40 BYTE),
MSG_PART NUMBER(2),
MESSAGE VARCHAR2(4000 CHAR),
CRE_DATE_TIME DATE
)

Sample data


INSERT INTO trail_log
(order_no, bs_name, message_sender,
message_receiver, transaction_id, msg_part,
MESSAGE,
cre_date_time
)
VALUES (123456789, 'business_service_name', 'name_of_application',
'receive_name', '2E2BE998F84833C7FB70189C60AEF58D', 1,
'<?xml version="1.0" encoding="UTF-8"?><soapenv:Envelope xmlns:soapenv=" http://schemas.xmlsoap.org/soap/envelope/" ><soapenv:Body>...</>',
TO_DATE ('08/14/2016 01:48:54', 'MM/DD/YYYY HH24:MI:SS')
);

DELETE /*+ PARALLEL(t,4)*/ FROM trail_log t WHERE cre_date_time<TRUNC(sysdate)-15;

There is a index as well on this table:

CREATE INDEX INDX_AUDIT_LOG_DATE ON TRAIL_LOG (CRE_DATE_TIME);

As this is online table I can't take the CTAS approach Create table ... select * from.. Rename is also not an option due to online table.

>> When I run the above DELETE it takes 15 min approx to delete 21,28,170 and sometimes due to continues sessions from front end applications (INSERTS), some locking happens (not sure now) due to which database is chocked and a total chaos :(

So can you suggest me any approach which can reduce the time taken while DELETE. I know that partition can one of the way and that I am surely considering as a long term solution. But like anything for short term?

//regards
Vipin

and Connor said...

Partitioning is *definitely* the way to go here. So consider that as soon as possible.

In the meantime, rather than doing a big reosurce-hungry delete on the table, perhaps do it more as a "trickle" so its less intrusive on your applications. Something like:

declare
  d date;
loop
  d := sysdate;
  loop
    delete /*+ index_asc(t) */ trail_log t 
    WHERE cre_date_time<TRUNC(sysdate)-15
    and rownum < 10000;

    exit when sql%notfound  or sysdate - d > 60 / 86400;
  end loop;
  commit;
  dbms_lock.sleep(30);
end loop;
end;


ie, delete batches of 10,000 rows for up to 1 minute, and then commit and sleep for 30 seconds. That can be left to run continuously. You can of course, adjust the numbers and times to suit what works best for you.

There is also the row archival option to consider

https://docs.oracle.com/database/121/VLDBG/GUID-5A76B6CE-C96D-49EE-9A89-0A2CB993A933.htm#VLDBG14154

but that would need greater investigation to see if it meets your need.

Rating

  (8 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

Bulk Delete

Vipin Vij, October 11, 2016 - 7:10 am UTC

Hi Connor/TOM
I tried the solution but it is not working. Running for more than 12 hours I had to kill it. Secondly as you mentioned I can not leave the solution keep running on its own. AS the underlying table is online I have to keep strict eye on the same or else chaos for my database and application.

So is there any other approach you can suggest: Like: Shall I use
delete /*+ INDEX_ASC(t) */ from trail_log t WHERE cre_date_time<TRUNC(sysdate)-56
instead if
DELETE /*+ PARALLEL(t,4)*/ FROM trail_log t WHERE cre_date_time<TRUNC(sysdate)-15;

Will this help.
Connor McDonald
October 11, 2016 - 9:06 am UTC

You should re-read my answer carefully.

It was *designed* to run forever. But it doesn't (shouldnt) clobber your system in doing so...

definitely partition

ghassan, October 11, 2016 - 9:30 am UTC

yes one good solution is partition (interval, choose the reliable range one month? one quarter?)) on CRE_DATE_TIME;

first use the dbms redef afer creating the new target partitioned table.

no more deletes but a drop old partitions

think to compress (high?) the old partitions

this is the solution you should go for.

additionnal

ghassan, October 11, 2016 - 9:34 am UTC

whilst decision to take about partitionnig and if insist to make a delete , consider the nologging option in addition to the above.

dbms_parallel_execute

Rajeshwaran, Jeyabal, October 11, 2016 - 11:29 am UTC

Since, we are looking for a short term fix, how about DBMS_PARALLEL_EXECUTE api?

Some thing like this.

rajesh@ORA12C> declare
  2     l_sql long;
  3  begin
  4     l_sql := q'| select min(rid) start_id, max(rid) end_id |' ||
  5                      q'| from ( |' ||
  6                      q'| select rowid rid, ntile(100) over(order by rowid) nt |' ||
  7                      q'| from big_table |' ||
  8                      q'| where created < to_date('01-Jan-2017','dd-mon-yyyy') |' ||
  9                      q'|      ) |' ||
 10                      q'| group by nt  |' ;
 11     dbms_parallel_execute.create_task(task_name=>'DEMO_TASK');
 12     dbms_parallel_execute.create_chunks_by_SQL(
 13             task_name=>'DEMO_TASK',
 14             sql_stmt=>l_sql,
 15             by_rowid=>true);
 16  end;
 17  /

PL/SQL procedure successfully completed.

rajesh@ORA12C> column status format a10
rajesh@ORA12C> select status,count(chunk_id)
  2  from user_parallel_execute_chunks
  3  where task_name ='DEMO_TASK'
  4  group by status ;

STATUS     COUNT(CHUNK_ID)
---------- ---------------
UNASSIGNED             100

1 row selected.

rajesh@ORA12C> declare
  2     l_sql long;
  3  begin
  4     l_sql := 'delete from big_table
  5             where rowid between :start_id and :end_id ';
  6     dbms_parallel_execute.run_task(
  7             task_name=>'DEMO_TASK',
  8             sql_stmt=>l_sql,
  9             language_flag=>dbms_sql.native,
 10             parallel_level=>2);
 11  end;
 12  /

PL/SQL procedure successfully completed.

rajesh@ORA12C> 


given this
....
Important here to note is that this is online table in which 24*7 insert statement are coming from front end application. ONLY INSERTS come from front end application.
....

upon insert, oracle will do row level exclusive lock and Table level shared lock, which means those "new" row that got inserted after this DELETE process got started, wont be picked up until the next run (of this cleanup/delete process)

PS: consider increasing the "parallel_level" parameter, based on the available resources and server loads.
Connor McDonald
October 12, 2016 - 3:36 am UTC

Good point, but I'm inferring from the post that the "database is chocked" is due to some sort of resource starvation, because (as you've noted), the deletes should not be locking out inserts (unless there's unique keys floating about).

So that's why I've opted for the trickle-feed delete... dbms_parallel_execute *might* be a good option...but it also might make things worse.

Use BULK COLLECT FORALL

Jeff Jacobs, October 11, 2016 - 10:26 pm UTC

Why not use BULK COLLECT/FORALL in a fixed loop? SELECT ROWID ... FROM ... FORALL DELETE ... WHERE ROWID = ...?

BULK COLLECT/FORALL - Ignore

Jeff Jacobs, October 11, 2016 - 10:41 pm UTC

My bad; I misread Connor's initial response, as I'm in the middle of cleaning up a SELECT/ FOR LOOP mess (but it's UPDATE, note DELETE :-)

loop not stopping

Harshini, November 27, 2018 - 11:35 am UTC

declare
d date;
loop
d := sysdate;
loop
delete /*+ index_asc(t) */ trail_log t
WHERE cre_date_time<TRUNC(sysdate)-15
and rownum < 10000;

exit when sql%notfound or sysdate - d > 60 / 86400;
end loop;
commit;
dbms_lock.sleep(30);
end loop;
end;

I am able to delete million of rows using this query, but once it reaches 0, loop is not stopping. Please tell me how to stop the loop and end the procedure.
how to stop looping?
Connor McDonald
November 28, 2018 - 12:52 am UTC

declare
  d date;
  l_full_exit boolean;
loop
  d := sysdate;
  loop
    delete /*+ index_asc(t) */ trail_log t 
    WHERE cre_date_time<TRUNC(sysdate)-15
    and rownum < 10000;
    l_full_exit := sql%notfound;
    exit when l_full_exit or sysdate - d > 60 / 86400;
  end loop;
  commit;
  exit when l_full_exit;
  dbms_lock.sleep(30);
end loop;
end;
/




Georgi, November 09, 2021 - 8:29 am UTC

Does the SQL in the parallel_execute solution that Rajeshwaran provided guarantees that no other ROWIDs will be included, but only the results from the WHERE clause ?

Quote:
"rajesh@ORA12C> declare
2 l_sql long;
3 begin
4 l_sql := q'| select min(rid) start_id, max(rid) end_id |' ||
5 q'| from ( |' ||
6 q'| select rowid rid, ntile(100) over(order by rowid) nt |' ||
7 q'| from big_table |' ||
8 q'| where created < to_date('01-Jan-2017','dd-mon-yyyy') |' ||
9 q'| ) |' ||
10 q'| group by nt |' ;
11 dbms_parallel_execute.create_task(task_name=>'DEMO_TASK');
12 dbms_parallel_execute.create_chunks_by_SQL(
13 task_name=>'DEMO_TASK',
14 sql_stmt=>l_sql,
15 by_rowid=>true);
16 end;
17 / "


I am confused about the range between min(rid) and max(rid). Can other rows (which should not be deleted) get inside ?
Connor McDonald
November 10, 2021 - 4:12 am UTC

You would still need to have the extra conditions in your processing, ie

delete from table
where rowid between ...
and created < to_date('01-Jan-2017','dd-mon-yyyy')

More to Explore

VLDB

If you are new to partitioning, check out Connor McDonald's introduction series here.

VLDB

Documentation set on VLDB and Partitioning.