A reader, February 19, 2005 - 6:02 pm UTC
Excellent!
select * from t when creating temporary table
a reader, February 19, 2005 - 6:11 pm UTC
Tom,
I am aware we should use the column names when creating table using CTAS (create table as )syntax. However, when we are building an intermediate(temporary) table to swap a partition of a table with data in the temporary table, could we use select * or we should use individual columns even when we know that both the tables have identical structures. I can understand that you use select * from t for just demos.
Thanks for your time
Regards
February 20, 2005 - 9:38 am UTC
if the select * gets the columns in the right order (i used create table as select more out of convenience than anything else) it would be safe -- since this table is created exactly ONCE (and it'll either get it right or nor and you would know right away)
handling the hash-partitioned ones
Alberto Dell'Era, February 19, 2005 - 6:43 pm UTC
> using a partitioned table with a single partition
Interesting addendum: if the original table is partitioned by hash (quite common in a dwh), you can make the mirror table SUBpartitioned by hash and everything works just dandy, eg
create table algnc_extract_umbox (
username varchar2(50) not null
)
partition by hash (username) partitions 32;
create table algnc_extract_umbox_stage (
username varchar2(50) not null
)
partition by range (username)
subpartition by hash (username) subpartitions 32
(
partition p_max values less than (maxvalue)
);
alter table algnc_extract_umbox_stage
exchange partition p_max
with table algnc_extract_umbox
including indexes
without validation;
Obviously you can swap roles - the staging table can be the single-partition one or vice-versa.
Alberto
Out of the box!!!
raul, February 19, 2005 - 10:14 pm UTC
Excellent! Thanks for taking the time to give a neat solution.
'redo size' came down to 24M ( from 145M )
The APPEND hint did not make a difference though.
SQL> select LOG_MODE, FORCE_LOGGING from v$database;
LOG_MODE FOR
------------ ---
ARCHIVELOG YES
SQL> create table t as select * from dba_objects where 1=2;
Table created
SQL> exit
SQL> insert into t select * from dba_objects;
36811 rows created.
SQL> @mystat
Enter value for 1: redo size
NAME VALUE
--------------- ----------
redo size 4474248
SQL> truncate table t;
Table truncated.
SQL> exit
SQL> insert /*+ APPEND */ into t select * from dba_objects;
36811 rows created.
SQL> @mystat
Enter value for 1: redo size
NAME VALUE
--------------- ----------
redo size 4492168
February 20, 2005 - 10:01 am UTC
what is will reduce is the "delete rowid" messages placed into UNDO and the REDO assocaited with that. It can make a difference depending on the size of the rows (if you have really wide rows, you might not notice this)
ops$tkyte@ORA9IR2> select LOG_MODE, FORCE_LOGGING from v$database;
LOG_MODE FOR
------------ ---
ARCHIVELOG YES
ops$tkyte@ORA9IR2> drop table t;
Table dropped.
ops$tkyte@ORA9IR2> create table t as select id from big_table.big_table where 1=0;
Table created.
ops$tkyte@ORA9IR2> @mystat "redo size"
ops$tkyte@ORA9IR2> set echo off
NAME VALUE
------------------------------ ----------
redo size 11441736
ops$tkyte@ORA9IR2> insert into t select id from big_table.big_table;
20000000 rows created.
ops$tkyte@ORA9IR2> @mystat2
ops$tkyte@ORA9IR2> set echo off
NAME V DIFF
------------------------------ ---------- ----------------
redo size 357189944 345,748,208
ops$tkyte@ORA9IR2> truncate table t;
Table truncated.
ops$tkyte@ORA9IR2> @mystat "redo size"
ops$tkyte@ORA9IR2> set echo off
NAME VALUE
------------------------------ ----------
redo size 357311544
ops$tkyte@ORA9IR2> insert /*+ append */ into t select id from big_table.big_table;
20000000 rows created.
ops$tkyte@ORA9IR2> @mystat2
ops$tkyte@ORA9IR2> set echo off
NAME V DIFF
------------------------------ ---------- ----------------
redo size 609519956 252,208,412
<b>so, lots of rows -- small rows -- 70% of the redo is generated (no undo)</b>
but again, function of the width of the row, using wider rows, this savings will drift to the zero point as the amount of data to truly be logged overwhelms the "delete rowid" messages placed into undo -- the biggested redo savings came from
a) not deleting
b) not maintaining the indices
Yes! Sir Tom
raul, February 20, 2005 - 12:31 pm UTC
I have 'really' learned and understood some cool stuff here. I sincerely appreciate your time and patience. THANK YOU!!!
Is merging an option?
SFT, February 21, 2005 - 2:58 am UTC
If the summary table has a relatively small delta, i.e. very few new rows have to be inserted and old rows to be deleted would it be an option to use MERGE statement in scenario approx. like this:
1. Generate unique sequence number or timestamp
2. MERGE new rows into existing summaries using just generated ID and taking care of of unnecessary updates
3. Delete old rows that have ID <> that current one
Then redo related to insert/delete would not be that large just because there were not that many INSERTs and DELETEs and therefore not that many index entries had to be created/deleted. On the other hand update related redo is the function of actual data that are updated.
Is that correct?
February 21, 2005 - 10:16 am UTC
if you are not updating -- just inserting and deleting, where does merge come into play?
"unnecessary updates"?
sorry, not following you on this one.
Merging example
SFT, February 21, 2005 - 2:41 pm UTC
Hello Tom,
What I meant is that in some situations it is needed to replace not all the data, but some portion of it, e.g. all rows within a given range of key column value. Then your truncate/exchage approach does not work (? at least requires some modifications and presumably more than one partition).
Here is an example with benchmarking. It shows that trancate/exchange is clearly superior (375K redo), but MERGE/DELETE (2.5M) is better then DELETE/INSERT (6.5M). BTW elapsed time is also better.
SYS@ORA9> create table t
2 (
3 c1 number constraint t_pk primary key
4 ,c2 varchar(100)
5 ,c3 number -- session id
6 );
Table created.
SYS@ORA9>
SYS@ORA9> -- DELETE/INSERT case
SYS@ORA9> truncate table t;
SYS@ORA9> insert into t
2 select * from
3 (select
4 ROWNUM c1
5 ,TO_CHAR(ROWNUM) c2
6 ,null c3
7 from user_objects
8 where rownum <= 10000
9 )
10 where c1 between 1 and 9990
11 ;
9990 rows created.
Elapsed: 00:00:02.00
SYS@ORA9> @mystat "redo size"
NAME VALUE
-------------------- ----------
redo size 25284744
SYS@ORA9>
SYS@ORA9> delete from t;
9990 rows deleted.
Elapsed: 00:00:04.09
SYS@ORA9>
SYS@ORA9> insert into t
2 select * from
3 (select
4 ROWNUM c1
5 ,TO_CHAR(ROWNUM) c2
6 ,null c3
7 from user_objects
8 where rownum <= 10000
9 )
10 where c1 between 11 and 10000
11 ;
9990 rows created.
Elapsed: 00:00:04.04
SYS@ORA9> @mystat2
NAME V DIFF
-------------------- -------------------- --------------------
redo size 31743616 6458872
SYS@ORA9>
SYS@ORA9> -- MERGE/DELETE case
SYS@ORA9> var session_id number;
SYS@ORA9>
SYS@ORA9> truncate table t;
Table truncated.
SYS@ORA9> exec :session_id := 1;
SYS@ORA9> insert into t
2 select * from
3 (select
4 ROWNUM c1
5 ,TO_CHAR(ROWNUM) c2
6 ,:session_id
7 from user_objects
8 where rownum <= 10000
9 )
10 where c1 between 1 and 9990
11 ;
9990 rows created.
Elapsed: 00:00:02.00
SYS@ORA9> @mystat "redo size"
NAME VALUE
-------------------- ----------
redo size 32513408
SYS@ORA9>
SYS@ORA9> exec :session_id := 2;
SYS@ORA9> merge
2 into t t1
3 using
4 (
5 select * from
6 (select
7 ROWNUM c1
8 ,TO_CHAR(ROWNUM) c2
9 from user_objects
10 where rownum <= 10000
11 )
12 where c1 between 11 and 10000
13 ) t2
14 on (t2.c1 = t1.c1)
15 when matched then
16 update set
17 t1.c2 = t2.c2
18 ,t1.c3 = :session_id
19 when not matched then
20 insert
21 values
22 (
23 t2.c1
24 ,t2.c2
25 ,:session_id
26 )
27 ;
9990 rows merged.
Elapsed: 00:00:04.08
SYS@ORA9>
SYS@ORA9> delete from t where c3 <> 2;
10 rows deleted.
Elapsed: 00:00:00.00
SYS@ORA9> @mystat2
NAME V DIFF
-------------------- -------------------- --------------------
redo size 34961168 2447760
SYS@ORA9>
SYS@ORA9>
SYS@ORA9> -- TRUNCATE/INSERT/ADD CONSTRAINT case
SYS@ORA9>
SYS@ORA9> @mystat "redo size"
NAME VALUE
-------------------- ----------
redo size 34961168
Elapsed: 00:00:00.00
SYS@ORA9> truncate table t;
Table truncated.
Elapsed: 00:00:00.01
SYS@ORA9> alter table t drop constraint t_pk;
Table altered.
Elapsed: 00:00:00.02
SYS@ORA9> insert /*+ append */ into t
2 select * from
3 (select
4 ROWNUM c1
5 ,TO_CHAR(ROWNUM) c2
6 ,null c3
7 from user_objects
8 where rownum <= 10000
9 )
10 where c1 between 11 and 10000
11 ;
9990 rows created.
Elapsed: 00:00:02.02
SYS@ORA9> alter table t add constraint t_pk primary key (c1);
Table altered.
Elapsed: 00:00:00.06
SYS@ORA9> @mystat2
NAME V DIFF
-------------------- -------------------- --------------------
redo size 35336896 375728
Thanks for your patience (and for mystat* scripts:)
February 21, 2005 - 3:54 pm UTC
ok, i did not understand the question which really was
a) should I delete existing rows and insert all as new
or
b) update existing data and delete ones I don't want
and you would find the answer is sometimes a) and sometimes b) depends on the volume
in 10g, you might be interested in the merge with the delete option -- you can insert new, update existing and if the updated row meets some criteria - delete it.
Superb solution
A reader, February 22, 2005 - 6:12 am UTC
A reader, March 10, 2005 - 5:05 pm UTC
Hi Tom,
In your original answer, why are you gathering stats on NEW_T and using those stats to set the stats on table T ?
You could have directly gathered the stats on table T after exchange partition operation(skipping the gather stats operation on NEW_T) ?
thanks
March 10, 2005 - 7:39 pm UTC
I figured you want the table in there with stats ASAP, so gather on the table that is not part of the big table first, slide it in and set the stats
Less time for the table was are sliding the data into to have 'wrong' stats.
Any performance benefit using Materialized views doing full refresh
A reader, March 11, 2005 - 4:20 pm UTC
Hi Tom,
Thanks for the reply.
Based on your method(initial post), we are going to rewrite the method by which we do Materialized View(MV) refresh.
We are going to replace all our MVs with ordinary tables. I dont think we have any specific gain by using MVs since all our refreshes are full refreshes.
Create partitioned nologging temporary table(not global temporary) with similar definition as original MV(now not MV but table) - this create is not inside the refresh procedure
a) Truncate the partitioned temporary table
b) insert into this temporary table using /*+ append */ hint using the select in the MV definition
d) compress this temporary table using alter ... move compress (since it is a dataware house and our queries are faster with compressed table)
e) Drop indexes on orginal table
f) use exchange partition between partitioned temporary table and the original table
g) create all the indexes on original table
h) gather stats on original table
During initial tests, the refresh time has almost halved
Now a few clarifications :-
a) I assume there is no performance benefit by using Materialized views compared to ordinary tables(for full refresh) - other than the ease of maintenance by using the standard packages available for their refresh/maintenance etc - am I right here ?
b) Instead of using a ordinary table as temporary table, will it help(performance wise) if we use a global temporary table instead ?
b) Is there any way of refreshing the MV using 'append' hint ?
thanks
March 12, 2005 - 8:22 am UTC
d) would not be necessary, insert /*+ append */ will load compressed
second a) well, you'll be missing out o MV rewrites.
b) you cannot.
b2) they already do, mv full refreshes do what you described already! trace it
A reader, March 12, 2005 - 11:51 am UTC
"would not be necessary, insert /*+ append */ will load compressed"
Even after doing 'insert /*+ append */ ...' , we are able to bring down the size of table still further by using 'alter table ...move compressed'.
"second a) well, you'll be missing out o MV rewrites."
Neither query rewrite nor fast refresh option worked for us for MVs - both due to complexity of the MV definition or maybe due to lack of knowledge - there simply is too many restrictions in Oracle for fast refresh and query rewrites
March 12, 2005 - 2:14 pm UTC
how much? does not make sense if the table you are loading into is COMPRESSED already.
give us some metrics, show how you measured and the table was compressed during the insert append correct?
A reader, March 12, 2005 - 11:54 am UTC
they already do, mv full refreshes do what you described already! trace it
Maybe for single MV refresh the above is true.
But if you do a refresh of MV using dbms_mview.refresh, along with some other dummy MV, it always does a delete and then an ordinary insert.
March 12, 2005 - 2:14 pm UTC
if you use mv groups -- yes, but if they can do the above - then they would by definition have no need of refresh groups so -- it is already doing what they are doing.
A reader, March 13, 2005 - 7:29 pm UTC
Basically we want the data to be available for querying. So the only time where it is not available(in the above method) is during the 'alter table ... exchange partition..' which anyway takes only a fraction of a second.
Whereas if you do the refresh of a single MV,(which is a truncate and then insert /*+ append */ ), data is not available to other users for quering during the insert /*+ append */ operation.
Truncate -- does not affect standby ?
Balasubramanian., March 21, 2005 - 5:17 am UTC
Tom
YOu have stated that
"truncate
is standby safe 100% it just updates the dictionary:"
could you please explain.
Thanks and regards
March 21, 2005 - 10:29 am UTC
well, it just updates the dictionary, that is logged and those updates are applied to the standby so the data disappears on the standby as well.
Great stuff
Randy, March 21, 2005 - 3:26 pm UTC
I was planning on implementing something similar to this, but I have multiple, related tables. How can this work for that situation? I then started to look at the MERGE command, since it is transactional. I couldn't see how to make the DDL (partitions) solution work for a set of tables. What do you think?
March 22, 2005 - 11:00 am UTC
DDL won't let you do N tables at a time.
If you need them refreshed "consistently", you'll be back to using SQL -- just like an MV refresh. A single table refresh:
truncate + insert /*+ append */
a multi-table refresh
delete + insert
delete + insert
delete + insert
....
execute immediate 'truncate table new_t';
A reader, March 22, 2005 - 10:37 am UTC
I do not know the requirements but using the truncate, there will be a moment in time when a different session select no data from the table. If you use delete then insert, the table will alway be populted.
15 execute immediate 'truncate table new_t';
16
March 22, 2005 - 11:03 am UTC
new_t is never queried by anyone.
so, not so.
Possible to have "table t logging / table new_t nologging"?
A reader, March 23, 2005 - 6:00 pm UTC
Tom, dumb question perhaps ...
If I create (partitioned) table "t" *logging* and (exchanged) table "new_t" *nologging* in an instance which is in archivemode (think: standby db), will "t" be in logging mode (AFTER the exchange has happened) again?
Can I really have the cake, or did I miss something here?
Thanks,
Tomasz
March 23, 2005 - 6:31 pm UTC
well, was new_t loaded unlogged? if so, standby will not have any data -- even after the exchange, for it.
Possible to have "table t logging / table new_t nologging"?
A reader, March 24, 2005 - 5:13 am UTC
Ok, that's clear (that was a stupid question, of course).
Let me put it in another way. Forget standby now and simply
assume archive logging mode:
1) If i load "new_t" unlogged and exchange with "t", after
that exchange any changes to "t" would be logged, right?
2) But that wouldn't make sense, since I would lose the
very value of archive logging. I couldn't recover from
media failure afterwards, since the rows that have been
loaded unlogged would be missing, correct?
Thank you,
Tomasz
March 24, 2005 - 8:52 am UTC
1) direct path operations would not be. conventional path operations are always. As this was in support of a "full refresh", there would be no modfications to the "new_t" partition, it wasn't a consideration in this example.
2) this is a "do it yourself mv", they reload it every 5 minutes. It wasn't applicable, they would just recreate it.
"table t logging / table new_t nologging"
A reader, March 24, 2005 - 6:15 pm UTC
Thanks Tom, I groked it :)
Mille grazie,
Tomasz
delete - /*+APPEND*/ combination
A Reader, April 07, 2005 - 9:23 am UTC
Tom,
I just want to make sure my understanding is correct: delete - /*+append*/ is not a good combinations because each time the insert would above the HWM and leave those deleted blocks empty. We cannot use the trucate because in case of the procedue fails we need to rollback the deleted table and use the old data at the end. Should we simply remove the /*+append*/?
Thanks.
April 07, 2005 - 11:08 am UTC
insert is the only DML (well, merge but only for the insert part of it) that append could apply to.
I would remove the append only because it might have someone question you in the future "you don't believe append actually works there do you" sort of stuff.
Index Creation
Andy, April 07, 2005 - 2:39 pm UTC
Tom,
Great solution, but I do not understand your comment about the constraint creation. Does this mean that the index was not dropped when the constraint was dropped ?
'and index it, the create index doesn't have to generate undo (we just rollback
the dictionary change -- standby "safe"'
Thank you for this site.
April 07, 2005 - 4:32 pm UTC
should have said "and constrain it, which will create an index in this case"
the add constraint is adding an index, and doesn't generate undo, just redo
A slight variation on the theme...
Mark Brady, August 29, 2005 - 4:18 pm UTC
Oracle 8i
Let's say we have a partitioned Table. It has a column called effective date which is the range partition key. Every day new data is added to the current month's partition. Data is stale IF, after two months, a given ID has been written with an effective date in the last two month. Most ID's are written at least once a month but some are not. I want to wring out the stale data from my older partitions.
So I would modify the procedure you defined above in the following way.
Instead of creating the table in advance and inserting the rows-to-keep into it. I would create the table using CTAS with the select defining the records which need to be saved.
Once I have the saved records in the new table, I index, gather stats as above, then swap the partition in the main table with the partition in the new table, then drop the new table.
Eventually the CTAS should select no records and then I know there is no data needing to be kept and just drop the partition outright.
Did I miss something?
reader
A reader, September 22, 2005 - 2:54 pm UTC
I know that
delete generates lot of undo
insert generates least undo
is it true that
insert generates lot of redo
delete generates least redo ( because to redo the change
is just to delete the rowid , so only information
required is the rowid to be deleted )
September 22, 2005 - 5:59 pm UTC
well, remember that indexes will affect this as well (they are undoable and logged as well)
but since UNDO is logged...... when you talk about redo, you have to talk about UNDO as well :)
reader
A reader, September 27, 2005 - 10:46 pm UTC
If the indexes are altered unusable/disable, will it still
generate the undo. The indexes can be rebuilt online after
the load
September 28, 2005 - 9:17 am UTC
if the indexes are not maintained, there is no undo to be generated for them.
redo for delete
steve, September 30, 2005 - 9:20 am UTC
Hi Tom,
I'm on an oracle course at the moment and I'd like to
clarify somnething on deletes.
Q1. In the case of a delete, if we ignore the UNDO portion
that is logged in the redo logs, then is it true that
the amount of redo logged for a delete is small (i.e.
just the rowid to delete)?
Q2. Would this be one of the reasons why we cannot use
the redo logs (and archived redo logs) to flashback
the database? i.e. we do not have the before image.
Also, I guess things like NOLOGGING also influence
the decision....
The reason I ask is that the instructor said the redo logs
could have been used to flashback the database but a design
decision was made to use a new structure (flashback logs).
Thanks
Steve
Just to clarify something In the case
September 30, 2005 - 11:05 am UTC
q1) forgetting about indexes, yes.
q2) well, in theory you can - because I cannot forget about the undo myself, it is there. But UNDO is used to UNDO things in Oracle.
technically, flashback database could be done with only redo, technically.
Thanks
Steve, September 30, 2005 - 11:09 am UTC
Thanks Man.
But then you would be dependent on the UNDO_RETENTION in that case - correct?
September 30, 2005 - 11:57 am UTC
if you used redo, you would be dependent on the amount of redo you had - undo would be in there.
but they don't so it is all hypothetical
DML's performance with partitioned table
Pablo Rovedo, September 13, 2006 - 8:58 am UTC
We have an application that insert/delete many times per second in a table. Because this table was generating too many archives i created a one partition table as you suggested and the archives generation was reduced. My doubt is if this change could affect the dml performance over the table (we have a RAC with 2 nodes, version 10.2.0.2).
Thanks in Advance