Skip to Main Content
  • Questions
  • Is there any faster way to perform merge on 120k records

Breadcrumb

Question and Answer

Chris Saxon

Thanks for the question, Priyadarshni.

Asked: April 23, 2020 - 9:37 am UTC

Last updated: April 27, 2020 - 4:26 pm UTC

Version: 11g

Viewed 1000+ times

You Asked

There are around 120k records in the database, and based on a few functions I calculate scores for all the records, weekly I have to update the table with new records and respective scores.

Below is a procedure that I am using to merge data into the table:

create or replace procedure scorecalc
AS
score1 number;
score2 number;
score3 number;
CURSOR cur IS
 SELECT Id_number from tableA;

        r_num cur%ROWTYPE;
BEGIN
  --OPEN cur;
  FOR r_num IN cur
  LOOP
    select functionA(r_num.id_number),functionb(r_num.id_number),functionc(r_num.id_number)  into score1, score2,score3 from dual;
Merge into scores A USING
(Select
 r_num.id_number as ID, score1 as scorea, score2 as scoreb, score3 as scorec, TO_DATE(sysdate, 'DD/MM/YYYY') as scoredate
FROM DUAL) B
ON ( A.ID = B.ID and A.scoredate = B.scoredate)
WHEN NOT MATCHED THEN
INSERT (
 ID, scorea, scoreb, scorec, scoredate)
VALUES (
 B.ID, B.scorea, B.scoreb, B.scorec,B.scoredate)
WHEN MATCHED THEN
UPDATE SET
 A.scorea = B.scorea,
 A.scoreb = B.scoreb,
 A.scorec = B.scorec;
 COMMIT;
  END LOOP;
END;


whereas function A/ B/ C has complex queries, joins in it to calculate the score.

Please suggest any way to improve the performance because currently with this snippet of code I am only able to insert some 2k records in 1 hour? Can I use parallel DML here? Thank you!

and Connor said...

1) remove the commit
2) remove the loop
3) remove the PLSQL

:-)

The whole code can be a single SQL

Merge into scores A 
USING
    (Select r_num.id_number as ID, functionA(r_num.id_number) as scorea, functionb(r_num.id_number) as scoreb, functionc(r_num.id_number) as scorec, TO_DATE(sysdate, 'DD/MM/YYYY') as scoredate
     FROM tableA r_num) B
     
ON ( A.ID = B.ID and A.scoredate = B.scoredate)
WHEN NOT MATCHED THEN
INSERT (
     ID, scorea, scoreb, scorec, scoredate)
    VALUES (
     B.ID, B.scorea, B.scoreb, B.scorec,B.scoredate)
WHEN MATCHED THEN
UPDATE SET
     A.scorea = B.scorea,
     A.scoreb = B.scoreb,
     A.scorec = B.scorec;



Rating

  (2 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

Thanks Connor for solution

A reader, April 24, 2020 - 5:58 am UTC

But I have one concern still it is taking 6 hours to merge all 120k records, can I use parallel DMLs here or should I consider rewriting my functions which has complex queries and joins?

Thanks for helping me here


Chris Saxon
April 27, 2020 - 4:26 pm UTC

6 hours to merge 120,000 rows is a long time! As "A reader" suggests below, get the plan for this statement, ensure all necessary indexes are in place, etc.

6 hrs for 120K records

A reader, April 24, 2020 - 6:49 am UTC


Have you tried the usual troubleshooting steps?

Get the plan or enable trace with waits and then tkprof it to get an idea on the bottleneck.

Cheers!

More to Explore

PL/SQL demos

Check out more PL/SQL tutorials on our LiveSQL tool.

PL/SQL docs

PL/SQL reference manual from the Oracle documentation library