Skip to Main Content
  • Questions
  • Bulk move data from tables to another, within and across databases

Breadcrumb

XMAS_TREE

The AskTOM team is taking a break over the holiday season, so we're not taking questions or responding to comments, but we might still sneak in a few Office Hours sessions! Thanks for being a member of the AskTOM community.

Question and Answer

Tom Kyte

Thanks for the question, jyo.

Asked: February 06, 2007 - 11:07 am UTC

Last updated: March 17, 2011 - 8:35 am UTC

Version: 10.2.0

Viewed 10K+ times! This question is

You Asked

Dear Tom,

Before posting the question, i would like to thank you for providing the very
best solutions to lots and lots of questions here.

I want to move (cut and paste) data from threes tables (say
table1,table2,table3) to another three tables(say
table1_live,table2_live,table3_live) in the database, where the status column in
table1 is not null. table1,table2, table3 has millions of records in it.

Sample table script

CREATE TABLE TABLE1( ID NUMBER,t11 VARCHAR2(100), STATUS VARCHAR2(10), t12
CLOB, t13 DATE);

ALTER TABLE TABLE1 ADD (CONSTRAINT PK_TABLE1 PRIMARY KEY (ID));

CREATE TABLE TABLE1_LIVE(ID NUMBER, t11 VARCHAR2(100),STATUS VARCHAR2(10),
t12 CLOB, t13 DATE);

CREATE TABLE TABLE2(EID NUMBER,ID NUMBER, t21 VARCHAR2(100), t22
VARCHAR2(100), t23 DATE);

ALTER TABLE TABLE2 ADD (CONSTRAINT PK1 PRIMARY KEY (EID));

ALTER TABLE TABLE2 ADD (CONSTRAINT FK1 FOREIGN KEY (ID) REFERENCES TABLE1 (ID));

CREATE TABLE TABLE2_LIVE(EID NUMBER, ID NUMBER, t21 VARCHAR2(100), t22
VARCHAR2(100), t23 DATE);

CREATE TABLE TABLE3(LID NUMBER,ID NUMBER,t31 VARCHAR2(100 ),t32
VARCHAR2(100 BYTE),t33 DATE);

ALTER TABLE TABLE3 ADD (CONSTRAINT PK2 PRIMARY KEY (LID));

ALTER TABLE TABLE3 ADD (CONSTRAINT FK2 FOREIGN KEY (ID) REFERENCES TABLE1
(ID));

CREATE TABLE TABLE3_LIVE(LID NUMBER,ID NUMBER,t31 VARCHAR2(100 ),t32
VARCHAR2(100 BYTE),t33 DATE);


What will be the fastest option to do this using procedure. this procedure will
be executed every day midnight.

In a similar way, i want to move data from table1_live,table2_live,table3_live
tables to similar tables in another database through db link.

i have already tried, 'insert into .. select *' option, and then do delete, but
the delete operation is taking too much time.

please help me in this.

Thanks & Regards

and Tom said...

insert into table1_live select * from table1 where status is not null;

do the same for table2 and table3.


you have the ability to do parallel deletes.

If the number of rows you are deleting is HUGE relative to the table itself, you could:

ops$tkyte%ORA10GR2> create table t1 as select * from scott.emp;

Table created.

ops$tkyte%ORA10GR2> create table t1_live as select * from t1 where 1=0;

Table created.

ops$tkyte%ORA10GR2> create table t1_temp as select * from t1 where 1=0;

Table created.

ops$tkyte%ORA10GR2>
ops$tkyte%ORA10GR2> insert
  2    when (comm is not null) then
  3      into t1_live( empno, ename, job, mgr, hiredate, sal, comm, deptno )
  4               values ( empno, ename, job, mgr, hiredate, sal, comm, deptno )
  5    when (comm is null) then
  6      into t1_temp( empno, ename, job, mgr, hiredate, sal, comm, deptno )
  7               values ( empno, ename, job, mgr, hiredate, sal, comm, deptno )
  8  select * from t1
  9  /

14 rows created.

ops$tkyte%ORA10GR2>
ops$tkyte%ORA10GR2> drop table t1;

Table dropped.

ops$tkyte%ORA10GR2> alter table t1_temp rename to t1;

Table altered.


split the table T1 into two (t1_live and t1_temp) and then drop T1 and just rename T1_TEMP (the rows to keep) to T1

Rating

  (3 ratings)

We're not taking comments currently, so please try again later if you want to add a comment.

Comments

table partition concept for data move

jyo, February 07, 2007 - 8:51 am UTC

Dear tom,

Thank you very much for the response. Using temp table insert and drop is not a feasible in my present scenario.

Instead of inserting into live table and deleting from first table, i tried to partition the tables as given below. So when i update the status column the data will be moved to the other partition.

Here the main issue is the partition key(status) value will be null when data is inserted into the table.
This value will be updated based on some conditions by a procedure every night.
I think the partition key updation is not recommended and the row movement will be an overhead.

My first question is

from the following options which one will be faster

1. inserting bulk data from one table to similar other table and deleting from first table

2. Partitioning the table and moving data from one partition to other

3. create the tables with one default partition on status column. update the status column through procedure. insert the data into similar live tables based on status. split the default partion into two. drop the new partition.
(for doing this i have to disable the constraints . i want to know, is this recommended to do the actions like split, drop, disable etc in a procedure which will be scheduled to run evey night)

My second question is

is there any performance improvement if we keep the data in two tables instead of same table with two partitions. suppose there are huge amount of data in one partition. will this cause any performance problem for query/update on the other partition with less data.

CREATE TABLE TABLE1( ID NUMBER,
t11 VARCHAR2(100),
STATUS VARCHAR2(10),
t12 CLOB,
t13 DATE)
PARTITION BY LIST (status)
(
PARTITION P1 VALUES (null),
PARTITION P2 VALUES ('ACT')
)
NOCOMPRESS
NOCACHE
NOPARALLEL
MONITORING
ENABLE ROW MOVEMENT;

CREATE TABLE TABLE2(EID NUMBER,
ID NUMBER,
t21 VARCHAR2(100),
t22 VARCHAR2(100),
t23 DATE,
STATUS VARCHAR2(10))
PARTITION BY LIST (status)
(
PARTITION P1 VALUES (null),
PARTITION P2 VALUES ('ACT')
)
NOCOMPRESS
NOCACHE
NOPARALLEL
MONITORING
ENABLE ROW MOVEMENT;


CREATE TABLE TABLE3(LID NUMBER,
ID NUMBER,
t31 VARCHAR2(100 ),
t32 VARCHAR2(100 BYTE),
t33 DATE,STATUS VARCHAR2(10))
PARTITION BY LIST (status)
(
PARTITION P1 VALUES (null),
PARTITION P2 VALUES ('ACT')
)
NOCOMPRESS
NOCACHE
NOPARALLEL
MONITORING
ENABLE ROW MOVEMENT;

ALTER TABLE TABLE1 ADD (CONSTRAINT PK_TABLE1 PRIMARY KEY (ID));

ALTER TABLE TABLE2 ADD (CONSTRAINT PK1 PRIMARY KEY (EID));

ALTER TABLE TABLE2 ADD (CONSTRAINT FK1 FOREIGN KEY (ID) REFERENCES TABLE1 (ID));

ALTER TABLE TABLE3 ADD (CONSTRAINT PK2 PRIMARY KEY (LID));

ALTER TABLE TABLE3 ADD (CONSTRAINT FK2 FOREIGN KEY (ID) REFERENCES TABLE1
(ID));
Tom Kyte
February 07, 2007 - 6:44 pm UTC

why isn't it feasible?

if you update a partition key and cause it to move, it will physically be the equivalent processing wise of DELETE + INSERT - because that is what happens under the covers.

Moving data across dblinks

Vicky, March 02, 2008 - 11:20 am UTC

Hi,

We are pulling data over dblink from 15 systems into our database i.e. 15 different source tables and 15 different target tables.We have once local index for each of the 15 target tables.

Every day we are suppose to receive 15 million records from each of the sub-system as per volumetrics.Therefore a total of 225 million records per day.

We have currently scheduled jobs every 4 hours per day so that the load gets divided.

Our approach to pulling data over dblink is as follows;
1) setting the index to unusable.
2) alter session set skip_unusable_indexes=true;
3) Pull data over dblink based on the criteria
4) rebuilding all the modified partitions i.e. only those partitions in which data got inserted.

We are working on oracle 10g.
My question are
1)whether the performance will degrade over a period of time as i am going to receive 15 million records every day for one sub-system.
2)Would the index be unusable when the data load is going on.
3)Anything I am missing out or can be done to improve performance.

The data is never going to be deleted from target tables.
Currently i dont have a big database to test on a huge volume,however is the benchmarking that i have done:

1) With append and parallel hint , it takes 2.5 minutes for 2 million records.
2) With append hint , it takes 2.5 minutes for 2 million records.
3) With nno hint , it takes 6.5 minutes for 2 million records.

One more question which is not related to the above scenario but curious to know:

What if I had a global index on the target table? Am I suppose to re-build the index after every load? Becauase I am going to receive 15 million records per day and over a period of time , rebuilding the index every 4 hours will degrade the performance? wont it ? Is there any other was for bulk insert on a table that has global index on it.

Regards
Tom Kyte
March 02, 2008 - 2:24 pm UTC

1) not really possible to say. It will not "degrade", but if you have to scan more and more and more information each time, of course it will take longer each time. It truly depends on the nature of your queries.

2) well, you set it unusable, it would be unusable before the load even began. You might consider using /*+ APPEND */ - and just maintaining the index. append will direct path load - and during that process, the newly loaded data will create a "mini index" for just the loaded data in temp, and then merge that in bulk back with the other index data efficiently after the process. In that fashion, you'll avoid a) writing the data and then b) reading the data in order to rebuild an index.


and I presume you meant "i'll set index partitions unusable". You are mixing together talk of tables and partitioning, not sure what you are doing.



as for your timings, that'll be a big "it depends" as well - for all I know on your test system, you had a cache on a san and the 2million records you wrote were all written to the cache in direct path mode and when you go "real" it'll flood the cache - so, you'll really want to test that out to scale on something resembling the production system.


about the global index - You might maintain the index anyway, be it local or global - append is pretty efficient doing that. Since you want to load and query at the same time (it sounds like) disabling the index maintenance probably is not in the cards. You'll want to maintain them.

viky, March 17, 2011 - 4:52 am UTC

i have 12 tables and i want to append table records in one big table one by one.
Tom Kyte
March 17, 2011 - 8:35 am UTC

great, run 12 inserts?

not sure what you are trying to ask or anything...

More to Explore

CLOB/BLOB/etc

Complete documentation on Securefiles and Large Objects here