Skip to Main Content
  • Questions
  • Bulk insert of more than 6 million every day in OLTP system

Breadcrumb

Question and Answer

Connor McDonald

Thanks for the question, Mike.

Asked: January 07, 2019 - 11:56 pm UTC

Last updated: January 10, 2019 - 1:08 am UTC

Version: Oracle 12.1.0.2

Viewed 10K+ times! This question is

You Asked

Hi Tom,

Following loop is my biggest problem in live system. Following loop will run every day to insert more than 6 million records in OLTP system and taking 20 hours. Memory wise we don't have any issues.

6TB of memory
195 CPU cores
2.5 GHz cpu speed
Intel Xeon E5-4640 CPU Type


Following is the code: Can you please help me how to speed up the following loop?

BEGIN
OPEN rollup_csr;
LOOP
FETCH rollup_csr BULK COLLECT INTO v_csr_tab LIMIT 100;
EXIT WHEN v_csr_tab.COUNT = 0;
BEGIN

FORALL i in v_csr_tab.FIRST..v_csr_tab.count save exceptions
INSERT INTO blahblah
(create_date, test1, test2, test3, test4,
test5, test6, test7,
test8, test9, test10, test11, test12)
VALUES
(v_run_date,
v_csr_tab(i).test1, v_csr_tab(i).test2,
v_csr_tab(i).test3, v_csr_tab(i).test4,
v_csr_tab(i).test5, v_csr_tab(i).test6,
v_csr_tab(i).test7, v_csr_tab(i).test8, v_csr_tab(i).test9,
v_csr_tab(i).test10 / v_cnt, v_csr_tab(i).test11 / v_cnt, (((v_csr_tab(i).test11 / v_cnt) / (v_csr_tab(i).test10 / v_cnt)) * 100)
);

EXCEPTION

WHEN dml_errors THEN
null;
WHEN OTHERS THEN
v_err_code := SQLCODE;
v_err_msg := SUBSTR(SQLERRM, 1, 250);
INSERT INTO error_table
(err_num, err_msg, module, create_dtm)
VALUES
(v_err_code, v_err_msg, 'Inserting - blahblah', sysdate);

END;
COMMIT;
END LOOP;

CLOSE rollup_csr;
END;
END;
/

Greatly appreciate.

Mike

and Connor said...

You don't have a code issue, you have a design or SQL performance issue.

Here's my laptop using pretty much your code, inserting 5million rows in 12 seconds

SQL> create table t as select s.* from dba_source s,
  2   ( select 1 from dual connect by level <= 10 );

Table created.

SQL>
SQL> select count(*) from t;

  COUNT(*)
----------
   5058450

1 row selected.

SQL>
SQL> create table t1 as select * from t where 1=0;

Table created.

SQL>
SQL> set timing on
SQL> declare
  2    cursor rollup_csr is select * from t;
  3    type row_list is table of t%rowtype index by pls_integer;
  4    v_csr_tab row_list;
  5  BEGIN
  6  OPEN rollup_csr;
  7  LOOP
  8    FETCH rollup_csr BULK COLLECT INTO v_csr_tab LIMIT 100;
  9    EXIT WHEN v_csr_tab.COUNT = 0;
 10    BEGIN
 11
 12      FORALL i in v_csr_tab.FIRST..v_csr_tab.count save exceptions
 13        INSERT INTO t1 values v_csr_tab(i);
 14    END;
 15  COMMIT;
 16  END LOOP;
 17
 18  CLOSE rollup_csr;
 19  END;
 20  /

PL/SQL procedure successfully completed.

Elapsed: 00:00:12.33


So your 20hours means either

- the cursor you are opening is a very expensive SQL to execute, or
- the insert target table has things which are killing the insert speed, eg too many indexes, triggers, etc

So add some timings to each phase, and perhaps do some tracing with:

dbms_monitor.session_trace_enable

and see where that time is being lost.

Your *code* is fine from a performance perspective (but also check out "DML error logging")



Rating

  (1 rating)

Is this answer out of date? If it is, please let us know via a Comment

Comments

Bulk insert

Partha, January 08, 2019 - 2:35 am UTC

Thank you Connor
Connor McDonald
January 10, 2019 - 1:08 am UTC

Glad we could help

More to Explore

PL/SQL demos

Check out more PL/SQL tutorials on our LiveSQL tool.

PL/SQL docs

PL/SQL reference manual from the Oracle documentation library