Skip to Main Content

Breadcrumb

Question and Answer

Tom Kyte

Thanks for the question.

Asked: July 06, 2001 - 12:49 pm UTC

Last updated: August 12, 2006 - 8:00 pm UTC

Version: 8.1.7

Viewed 1000+ times

You Asked

Tom

We have Table A, into which we insert a million records everyday, after deleting previous day records.( we have other tables where we do the same thing except that the interval of time is weekly and monthly). Now our requirement has changed and we need to keep 10 days data . But everyday we should delete the olders day's data and add the current days data.

We have decided that partitioning the table is the way to go. Can you please discuss the sql script which can make the whole process automatic, that everyday a new partition gets added, and the oldest partition is dropped.

Could you also discuss any other issues which you can foresee with the solution we are trying to arrive at..

Thank you

and Tom said...

Here is an example of s sliding window:

tkyte@TKYTE816> CREATE TABLE partitioned
2 ( timestamp date,
3 id int
4 )
5 PARTITION BY RANGE (timestamp)
6 (
7 PARTITION fy_1999 VALUES LESS THAN
8 ( to_date('01-jan-2000','dd-mon-yyyy') ) ,
9 PARTITION fy_2000 VALUES LESS THAN
10 ( to_date('01-jan-2001','dd-mon-yyyy') ) ,
11 PARTITION the_rest VALUES LESS THAN
12 ( maxvalue )
13 )
14 /
Table created.

tkyte@TKYTE816> create index partitioned_idx_local
2 on partitioned(id)
3 LOCAL
4 /
Index created.

So, this sets up our 'warehouse' table. The data is partitioned by fiscal year
and we have the last two years worth of data online. Notice that I left an empty
partition THE_REST at the end of the table. This will facilitate sliding new
data in quickly. Now, it is the end of the year and we would like to:

1. Remove the oldest fiscal year data. We do not want to lose this data forever,
we just want to age it out and archive it.
2. Add the newest fiscal year data. It will take a while to load it, transform
it, index it, and so on. We would like to do this work without impacting the
availability of the current data if at all possible.

The steps I might take would be:

tkyte@TKYTE816> create table fy_1999 ( timestamp date, id int );
Table created.

tkyte@TKYTE816> create index fy_1999_idx on fy_1999(id)
2 /
Index created.

tkyte@TKYTE816> create table fy_2001 ( timestamp date, id int );
Table created.

load fy-2001-here

tkyte@TKYTE816> create index fy_2001_idx on fy_2001(id) nologging
2 /
Index created.

What I've done here is to set up an empty 'shell' table and index for the oldest
data. What we will do is turn the current full partition into an empty partition
and create a 'full' table, with the FY 1999 data in it. Also, I've completed all
of the work necessary to have the FY 2001 data ready to go. This would have
involved verifying the data, transforming it – whatever complex tasks you need
to undertake to get it ready.

Now we are ready to update the 'live' data:

tkyte@TKYTE816> alter table partitioned
2 exchange partition fy_1999
3 with table fy_1999
4 including indexes
5 without validation
6 /
Table altered.

tkyte@TKYTE816> alter table partitioned
2 drop partition fy_1999
3 /
Table altered.

That's it for aging the old data out. We turned the partition into a full table
and the empty table into a partition. This was a simple data dictionary update –
no large amount of IO took place, it just happened. We can now export that table
(perhaps using a transportable tablespace) out of our database for archival
purposes. We could re-attach it quickly if we ever need to.

Next, we want to slide in the new data:

tkyte@TKYTE816> alter table partitioned
2 split partition the_rest
3 at ( to_date('01-jan-2002','dd-mon-yyyy') )
4 into ( partition fy_2001, partition the_rest )
5 /
Table altered.

tkyte@TKYTE816> alter table partitioned
2 exchange partition fy_2001
3 with table fy_2001
4 including indexes
5 without validation
6 /
Table altered.

That's it for sliding the data in. Again, this was instantaneous – a simple data
dictionary update. Splitting the empty partition takes very little real-time
since there never was, and never will be data in it. That is why I placed an
extra empty partition at the end of the table, to facilitate the split. Then, we
exchange the newly created empty partition with the full table and the full
table with an empty partition. The new data is online.


So, that is how to slide off old data, slide in new data -- very fast, no deletes.


You can definitely script that as well.

A big issue you might have to watch out for -- make sure you use LOCALLY partitioned indexes only. That means no unique constraints across partitions that don't include the TIMESTAMP column. Otherwise you have to rebuild th global indexes after the partition operations.



Rating

  (13 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

Fantastic

A reader, July 06, 2001 - 3:46 pm UTC

Very good solution and easy to understand.

Very userful information. Thanks to Tom.

Rao Munaga, July 24, 2001 - 4:02 pm UTC


very good ! but i want to konw

a reader, September 03, 2001 - 9:50 pm UTC

if every partitin in different tablespace , when you split
the partition ,you need set up new tablespace at first
can you create tablespace automatic?

very good ! but i want to konw

a reader, September 03, 2001 - 10:03 pm UTC

if every partitin in different tablespace , when you split
the partition ,you need set up new tablespace at first
can you create tablespace automatic?

Tom Kyte
September 07, 2001 - 12:19 pm UTC

There are never any tablespaces that are created implicitly -- you must explicitly create them.

Simply SUPERB

Narayanan.S.S., September 08, 2001 - 4:18 am UTC

your solutin is just fabulous. It looks nice too. It helped me to solve my hassle also Thank you.

very good answer but i want to know more about updating...

saggi, September 13, 2001 - 9:01 am UTC

i want to give rights on update(row) to user on a
particular partition how can i do it ????
thank ...

Tom Kyte
September 13, 2001 - 11:22 am UTC

You could use fine grained access control. See
</code> http://asktom.oracle.com/~tkyte/article2/index.html <code>

Or a trigger that inspected the :new/:old values and prevent users from updating rows they should not.

Or a view

Or a procedure to encapsulate the update logic

Grants are done at the OBJECT level (table in this case), not on individual partitions. For something like that, you will use one of the techniques above


Excellent answer

G.V.N., March 02, 2003 - 11:35 pm UTC


Re-attach old data

Michael, September 18, 2003 - 7:56 am UTC

Hello Tom,

I experimented with partitioning and also tried to reattach old data. I would like to know, if my approach is OK.

The setup:

SQL> CREATE TABLESPACE "MYDATA_2000" 
  2      LOGGING 
  3      DATAFILE 'E:\APPS\ORACLE\ORADATA\ORA92DB\MYDATA_2000_01.DBF'
  4      SIZE 1M EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT  
  5      AUTO ;

  1  CREATE TABLESPACE "MYDATA_2001"
  2      LOGGING
  3      DATAFILE 'E:\APPS\ORACLE\ORADATA\ORA92DB\MYDATA_2001_01.DBF'
  4      SIZE 1M EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT
  5*     AUTO

  1  CREATE TABLESPACE "MYDATA_2002"
  2      LOGGING
  3      DATAFILE 'E:\APPS\ORACLE\ORADATA\ORA92DB\MYDATA_2002_01.DBF'
  4      SIZE 1M EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT
  5*     AUTO

  1  CREATE TABLESPACE "MYDATA_2003"
  2      LOGGING
  3      DATAFILE 'E:\APPS\ORACLE\ORADATA\ORA92DB\MYDATA_2003_01.DBF'
  4      SIZE 1M EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT
  5*     AUTO

SQL> CREATE TABLE MYDATA
  2  (TIMESTAMP DATE, DATA VARCHAR2(10))
  3  PARTITION BY RANGE (TIMESTAMP)
  4  (  PARTITION MYDATA_2000
  5       VALUES LESS THAN (TO_DATE('01.01.2001', 'dd.mm.yyyy'))
  6       TABLESPACE MYDATA_2000,
  7     PARTITION MYDATA_2001
  8       VALUES LESS THAN (TO_DATE('01.01.2002', 'dd.mm.yyyy'))
  9       TABLESPACE MYDATA_2001,
 10     PARTITION MYDATA_2002
 11       VALUES LESS THAN (TO_DATE('01.01.2003', 'dd.mm.yyyy'))
 12       TABLESPACE MYDATA_2002,
 13    PARTITION MYDATA_2003
 14       VALUES LESS THAN (TO_DATE('01.01.2004', 'dd.mm.yyyy'))
 15       TABLESPACE MYDATA_2003
 16  )
 17  /

SQL> INSERT INTO MYDATA(TIMESTAMP, DATA)
  2  VALUES (TO_DATE('01.06.2000', 'dd.mm.yyyy'), 'DATA2000');

  1  INSERT INTO MYDATA(TIMESTAMP, DATA)
  2* VALUES (TO_DATE('01.06.2001', 'dd.mm.yyyy'), 'DATA2001')

  1  INSERT INTO MYDATA(TIMESTAMP, DATA)
  2* VALUES (TO_DATE('01.06.2002', 'dd.mm.yyyy'), 'DATA2002')

  1  INSERT INTO MYDATA(TIMESTAMP, DATA)
  2* VALUES (TO_DATE('01.06.2003', 'dd.mm.yyyy'), 'DATA2003')

SQL> SELECT * FROM MYDATA;

TIMESTAM DATA
-------- ----------
01.06.00 DATA2000
01.06.01 DATA2001
01.06.02 DATA2002
01.06.03 DATA2003

Now i want to archive the data from year 2000:

SQL> CREATE TABLE MYDATA_2000
  2  TABLESPACE MYDATA_2000
  3  AS
  4  SELECT * FROM MYDATA
  5  WHERE 1=0;

SQL> ALTER TABLE MYDATA
  2  EXCHANGE PARTITION MYDATA_2000
  3  WITH TABLE MYDATA_2000
  4  /

SQL> SELECT COUNT(*) FROM MYDATA;

  COUNT(*)
----------
         3

SQL> ALTER TABLE MYDATA
  2  DROP PARTITION MYDATA_2000;

SQL> ALTER TABLESPACE MYDATA_2000 OFFLINE;

Now i zip and archive the dbf file(s) of this tablespace (MYDATA_2000).

To reattach i do the following:

Dearchive and unzip the dbf file(s) from this tablespace to the right location. Then:

SQL> ALTER TABLESPACE MYDATA_2000 ONLINE;

SQL> SELECT * FROM MYDATA_2000;

TIMESTAM DATA
-------- ----------
01.06.00 DATA2000

SQL> ALTER TABLE MYDATA
  2  SPLIT PARTITION MYDATA_2001
  3  AT (TO_DATE('01.01.2001', 'dd.mm.yyyy'))
  4  INTO (PARTITION MYDATA_2000 TABLESPACE MYDATA_2000,
  5        PARTITION MYDATA_2001 TABLESPACE MYDATA_2001)
  6  /

SQL> ALTER TABLE MYDATA
  2  EXCHANGE PARTITION MYDATA_2000
  3  WITH TABLE MYDATA_2000;

SQL> SELECT * FROM MYDATA;

TIMESTAM DATA
-------- ----------
01.06.00 DATA2000
01.06.01 DATA2001
01.06.02 DATA2002
01.06.03 DATA2003

SQL> DROP TABLE MYDATA_2000;

It seems to work, but is this the right way to re-attach old data??

(To get the data from 2004 i do:

  1  ALTER TABLE MYDATA
  2  ADD PARTITION MYDATA_2004
  3      VALUES LESS THAN  (TO_DATE('01.01.2005','dd-mm-yyyy'))
  4*     TABLESPACE MYDATA_2004
SQL> /
)

Thanks. 

Tom Kyte
September 18, 2003 - 10:40 am UTC

that certainly works. another option is to just let the empty partition "stay" there -- don't drop it, just keep it.

less work during reattach time.

parallel

alex, December 01, 2003 - 4:12 pm UTC

Tom

insert /*+ append parallel(xyz,8) */ into xyz select /*+ parallel(xyz1,8) */ * from xyz1;

would the above sql be faster or inserting 100,000 records in a loop through pro C woould be faster for 3M records.

thanks

Tom Kyte
December 02, 2003 - 8:13 am UTC

does not compute.

parallel 8 for a measely 100k records would probably take longer to set up PQ then to actually insert the records.

but let me say this -- if you can:

a) insert into t select * from t2

or

b) open cursor on t2, fetch from t2 insert into t

you would choose (A) hands down, no real question about it.

Insertion Taking Long time

Prasanth PV, August 03, 2006 - 6:21 am UTC

Hi Tom,
Thanks for your reply.

Currently Our application is inserting into 4 tables sequentially, having minimum 600 columns each.

Some times the insertion is taking long time ( more than 10 seconds for 1 record) for all insertion into all tables.

How to tune this insertion statement/database, to make it faster.

Please help on this.

With Regards
Prasanth Pv

Tom Kyte
August 03, 2006 - 9:36 am UTC

trace it, tkprof it and see what it is waiting on.

Insertion Taking Long time

Prasanth PV, August 03, 2006 - 6:22 am UTC

Hi Tom,
Thanks for your reply.

Currently Our application is inserting into 4 tables sequentially, having minimum 600 columns each.

Some times the insertion is taking long time ( more than 10 seconds for 1 record) for all insertion into all tables.

How to tune this insertion statement/database, to make it faster.

Please help on this.

With Regards
Prasanth Pv

row chaining

Prasanth, August 11, 2006 - 1:04 am UTC

Hi Tom,
Whether row chaning will reduce the performance of insert?
Regards
Prasanth.PV

Tom Kyte
August 11, 2006 - 10:38 am UTC

row chaining happens when you UPDATE a row and it migrates OR when the row is too large to fit on a block and you INSERT it.

Since that is unavoidable in the case of the insert, it doesn't reduce the performance of anything - it is the base measure of the performance of the insert.




Very need full

cnu, August 12, 2006 - 9:27 am UTC

Hi Tom ,

Thanks for your Help for the oracle world.

We have two databases(old and new) ,the old one is oracle 8.1.7.0.0 on HP TRU64 and the new one is oracle 10.2.0.1.0 on IBM AIX.We are
Planning to migrate the data from old machine to new machine. we have tables partitioned(30 partitions) with each partition having 2 to 3 million records.
We have decided to do this migration using DB Links.We are creating the DB links from HP to IBM and inserting the data into IBM from HP.
We are selecting the data partition wise and inserting into IBM machine.
We are testing this as mentioned below.

database1:HP,
database2:IBM
table:table1(30 partitions, where we kept 16 rows in two different partitision,toal 32 rows)
dblink from HP to IBM :db_link_1
--executing from HP machine.

declare
cursor c1(t varchar2) is select partition_name from user_partitions where table_name=t1 order by partition_name;
begin
for i in c1
loop
execute immediate 'insert into table1@db_link_1 select * from table1 partition ('||i.partition_name||')';
commit;
end loop;
end;

After the execution , in the IBM SDP we are having 1024 rows.i.e 32 rows(all the rows in the HP table) getting inserted for each partition.i.e 32*32 .
It is not selecting the rows from a particular partition.
Could you please let me know why this is happening........

Could you also discuss any other issues which you can foresee with the solution we are trying to arrive at..


Tom Kyte
August 12, 2006 - 8:00 pm UTC

should be 10.2.0.2 shouldn't it....



which partition? is there something funny in the partition naming? why don't you print out the partition names as you fetch from them (for debugging?)


also, maybe you want to set up a script that you pass the partition name and parition key ranges into into:


insert /*+ append */ into table1 partition (partition_name)
select * from table1@db_link_1 where partition_key >= "lo val" and partition_key < "hi val";


you could then achieve some pretty good parallelism by running many sqlplus sessions at the same time.

More to Explore

VLDB

If you are new to partitioning, check out Connor McDonald's introduction series here.

VLDB

Documentation set on VLDB and Partitioning.