Skip to Main Content
  • Questions
  • UNDO generated By Index blocks during INSERT

Breadcrumb

May 4th

Question and Answer

Connor McDonald

Thanks for the question, Kabi.

Asked: January 10, 2017 - 11:55 am UTC

Last updated: November 07, 2023 - 5:43 am UTC

Version: 11.2.0.4.0

Viewed 10K+ times! This question is

You Asked

Hi, I am using version- 11.2.0.4 of oracle.

I have a query running and its getting slow down during specific time period of the day(10PM to 9AM) which i can also verify from the data present in DBA_HIST_SQLSTAT. The number of rows generated from the query, decreased from ~25million to ~15k during those snap duration(we have ~1hrs snap duration). Now going through the details in DBA_HIST_ACTIVE_SESS_HISTORY, i see the CURRENt_OBj# continuously showing as '0' during the slow period, while during good throughput time it was populated with valid value in CURRENt_OBj#. So i believe, it was doing UNDO read during the slow period, causing the slowness. Then from the real time monitor plan i am seein ~65% of the time has spent in the "index range scan" of "IDX1" and ~35% of the time has been spent on "TABLE ACCESS BY LOCAL INDEX ROWID" of same table, so i understand its doing mostly the UNDO read for the INDEX BLOCKS. and we have this table mostly INSERT only, we never/rarely perform DELETE/UPDATE on this. This query runs ~5hr+, but during UNDO read it sometime runs for 10+ hours and complete and sometimes fail with "snapshot too old".

So i tested the scenario as below , which will replicate our case, during SELECT, we have JAVA batch which pushes records into this table in chunks(~10000) and commits them, it does conventional path insert. Actually this batch job runs 24/7 and read from the file and push record into this table. Daily we have ~150million records getting pushed into this table.

What i observed, if during the SELECT running , there is INSERTS running around(without commit or gets committed afterwards), may be before/after the SELECT query starts, but in any cases it reads huge amount of UNDO blocks(possibly INDEX blocks).

1)So is my understanding is correct that its the INDEX blocks which generate such amount of UNDO and any SELECT during that period doing index scan, will get affected by this?
2)Is the only solution is to avoid the long running the SELECT during the INSERT period? or is ther any other solution(considering the SELECT query is tuned to its best)?


------------------------------ test case--------------------------

create table t1_undotest as select * from all_objects where 1=0;

-- initially filled the table with some record

Insert into t1_undotest select * from all_objects; - (it inserts 55897 records in one shot)
Insert into t1_undotest select * from all_objects;
Insert into t1_undotest select * from all_objects;
Insert into t1_undotest select * from all_objects;
Insert into t1_undotest select * from all_objects;
Insert into t1_undotest select * from all_objects;
Insert into t1_undotest select * from all_objects;
commit;


SCENARIO-1-
--------------
Now in session-1 , i executed below statement
SELECT * from t1_undotest; (I executed with SET AUTOTRACE TRACEONLY EXPLAIN STATISTICS, as i didnt need to results to be printed aall over)

While above query in session-1 was getting executed, I inserted record into same table from session-2 using below

Insert into t1_undotest select * from all_objects;
Insert into t1_undotest select * from all_objects;
Insert into t1_undotest select * from all_objects;
Insert into t1_undotest select * from all_objects;
Insert into t1_undotest select * from all_objects;
Insert into t1_undotest select * from all_objects;
commit;

Then i checked the UNDO records visited by the SELECT statement by executing it from SESSION-1(After the SELECT gets complted)

select 'undo: '||value from v$mystat natural join v$statname where name = 'data blocks consistent reads - undo records applied';

I see, its giving around ~8bytes UNDO.


SCENARIO-2-
-------------
Then i have tested another case,

In session-1 execute below statement (Do not commit)
Insert into t1_undotest select * from all_objects;
Insert into t1_undotest select * from all_objects;

In Session-2 executed below statement
SELECT * from t1_undotest; (I executed with SET AUTOTRACE TRACEONLY EXPLAIN STATISTICS, as i didnt need to results to be printed aall over)
Then i checked the UNDO records visited by the SELECT statement by executing it from SESSION-1(After the SELECT gets complted)
select 'undo: '||value from v$mystat natural join v$statname where name = 'data blocks consistent reads - undo records applied';

I see, its giving around ~4900 bytes UNDO.

SCENARIO-3
-----------
CREATE INDEX on OBJECT_TYPE column:
create index idx1_test on t1_undotest(object_name);
Now repeat SCENARIO-1, this time just to let the query traverse through INDEX, i have executed below in session-1

select /*+index(t1_undotest IDX1_TEST)*/ * from t1_undotest;

What i observed , its visiting ~218904 bytes of UNDO.


SCNERAIO-4
------------
I repeated SCENARIO-2, with the forced index hint, i observed, its visiting ~79154 bytes UNDO.

SCENARIO-5
-------------
In session-1 execute below statement ( commit)

Insert into t1_undotest select * from all_objects;
Insert into t1_undotest select * from all_objects;
Commit;

In Session-2 executed below statement
select /*+index(t1_undotest IDX1_TEST)*/ * from t1_undotest; (I executed with SET AUTOTRACE TRACEONLY EXPLAIN STATISTICS, as i didnt need to results to be printed aall over)

Then i checked the UNDO records visited by the SELECT statement by executing it from SESSION-1(After the SELECT gets complted)
select 'undo: '||value from v$mystat natural join v$statname where name = 'data blocks consistent reads - undo records applied';

I see, its giving around ~0 bytes UNDO.

and Connor said...

OK, I'm going to use a metaphor of a "technical book" here.

Let's say I write a technical book on SQL and PL/SQL features in 11g with say 300 pages. (That's my committed transactions).

I then draft up another 50 pages to update the book for the new 12c features (ie, uncommmited transactions). Someone coming along and reading the book can *easily* 'undo' those 50 pages because I dont even need to look at the contents of each page - as soon as I see that these are new pages for 12c, I can simply ignore them.

So there's a *little* bit of work to be done, but these new pages are all unrelated to the existing pages.

But now consider that my tech book has a typical "index of topics" at the back of the book. As I'm adding my 50 pages, I cant just add entries to the "end" of the index - they have to be scattered through the index in exactly the right sequence ("Associative Arrays" at the start, ie, leading "A", "ZERO_DIVIDE" at the end, ie, leading "Z").

So *now* a read (using that index) is a big deal - I have to resurrect the index as it was without those 50 pages worth of new index entries interspersed all over the place throughout that index. It's a big deal !

So to answer your questions

1)So is my understanding is correct that its the INDEX blocks which generate such amount of UNDO and any SELECT during that period doing index scan, will get affected by this?.

Yes.

2)Is the only solution is to avoid the long running the SELECT during the INSERT period? or is ther any other solution(considering the SELECT query is tuned to its best)?

Avoidance works. So does avoiding using the index if possible. For example, a large scale index range scan might be better suited to use a full scan anyway.

Rating

  (6 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

A reader, January 22, 2017 - 10:07 am UTC

Thank you so much. Really appreciate your help. Just a thought come into my mind, so isn't it possible to avoid that huge amount of UNDO read due to index blocks, if the inserts from JAVA batch can be made direct path(using append hint), rather conventional path? but yes although it seems tough deal as they would have to do it all in serial then and they wont be able to do it(inserts) in parallel.
and currently all the index are local partitioned on this big table, if making those global would help any way?
Connor McDonald
January 23, 2017 - 3:14 am UTC

If you do a direct mode insert, then the table is locked for the duration of the insert. If you can cope with that, then you avoid the undo issue to some degree.

But the problem exists predominantly during the period of uncommitted transactions, or long running queries that commenced before such transactions started. If you shrink their runtime, then you'll be less imapcted.

Question on similar line

Kabi P, February 03, 2017 - 2:18 pm UTC

Related to this, I have few questions , As i mentioned,We have this table gets loaded(only Insert) in conventional method through out the days 24/7, from different sessions Parallel y. and its Partitioned daily, and one Partition holds avg ~150million records daily. and the readers(rePorting queries) gets slowdown due to the effect of the UNDO reads sometimes, (mainly reading Index blocks). So i have below questions

1) If we make the table Partition hourly rather daily, will it gonna any(Positive/Negative) way affect the read/write Performance on this table and any way UNDO read will be better?
2)If we create an additional table(Tab1), which will be loaded additionally from this main table(Tab) with some other tool(like Golden gate), rather coneventional method of insert using INSERT statement, and rePorting queries will read the new table (Tab1), will it gonna helP us improving the read Performance?

3)In a general sense, is there any benefit of decreasing the Partition range(like from daily to hourly) for table, aPart from maintenance overhead.. ? how will be read and writes gets imPacted? if any hot block issue(or if we see lots of read by other session waits on this object) gets addressed by this?
Connor McDonald
February 03, 2017 - 7:42 pm UTC

insertion frequency/volume is not the issue when it comes to undo - it is the length of transactions, or the length of queries running around those transactions.

A reporting query that runs for 2 mins, by definition, needs to undo at most, 2 mins worth of transactions (committed or uncommitted). A query that runs for 2 hours might need to undo 2 hours of transactions.

The amount of undo work required is proportional to the query length (and the activity that occurs during that time). So you need to

a) reduce query execution times, and/opr
b) reduce transaction activity during those query execution times

Kabi P, February 03, 2017 - 8:49 pm UTC

Thank you for the resPonce. The only transaction which run during the SELECT runs is the INSERT and it runs 24/7 on this table. so as Per my second question, if we will load another new table using goldengate realtime rePlication and rePorts will read that new table, does it also generate/read same amount of undo? my assumPtion was that, it would transfer the logs, so will not generate UNDO, correct me if wrong?
And my first question was, because dev team was saying to decrease the granularity from Partition hourly to daily, As Per them, as write will be always haPPen in a new hourly Partition, so read/write overlaP would be minimized while reading current day Partition data from the table. Is it correct? and my last question was, if we will forget about the UNDO issue, in a general sence, if we decrease the granularity of Partition daily to hourly, then will it helP anything better read/write, for same data set?

Connor McDonald
February 06, 2017 - 2:56 pm UTC

Reading the redo does not create undo, but dont forget that GG ultimately resurrects the same DML on the target...and DML = undo.

As I said before, what matter is:

a) reduce query execution times, and/opr
b) reduce transaction activity during those query execution times

If changing your partitioning assists with either of those, then you'll see a benefit. If it doesn't, then it wont make a difference. A query that runs for 20mins against an hourly partitioned table will need to potentially undo 20 mins worth of activity, just like it would against a daily partitioned one.

Kabi P, February 08, 2017 - 3:11 pm UTC

Thank you so much for the response. Regarding you second point "b) reduce transaction activity during those query execution times ", is not this achievable by changing the partition strategy from daily to hourly? In current scenario, if we talk about the read/write simultaneous effect wrt to UNDO blocks generated by index, in case of write from different sessions(which is happening for the current hour data i.e 10 pm), the single index structure associated with daily partition will be updated and so if a SELECT query is running at same time scanning the daily partitioned data(12 Am to 9pM) using the index, it has to visit the same index blocks/UNDO blocks, even its not scanning the 10pm data. But in case of hourly partition/subpartition the index which would be getting updated by the parallel inserts would be the current one(that is associated with the 10pM hourly subpartition only), other 9-index structures(for 12AM to 9pm hourly partition) would be untouched, so the select query will have to revisit lesser UNDO. Is this correct understanding?
Connor McDonald
February 10, 2017 - 5:21 pm UTC

You're assuming a strictly *local* index policy here (which is fine if you can achieve that).

Kabi P, April 19, 2017 - 11:28 am UTC

Hi, i have a another question on same line, (apart from reducing SELECT query execution time, ) if we will not be able to make the insert(FROM Java) as direct path(due to locking issue and all), but if we will make it bulk like ('Insert into tab1.. select from ..) which will insert all records into tab1, once in half an hour or may be twice( rather in each minute/second which is happening now with row by row insert that to in parallel threads). Then i believe there would be improvement in execution time of the INSERT query from java side, But will it get better in terms of UNDO read for these long running SELECT queries which runs during that time?

In your reply as you said "insertion frequency/volume is not the issue when it comes to undo - it is the length of transactions", so does it mean that, as currently Java is executing row by row inserts and are committing a chunk of ~1000 inserts at a time and are doing it each minute/second(as and when we get input file from customer) through out the day and that to in multiple concurrent session/threads also. So by making it bulk, if they will just dump ~5million records at one shot, once in a hour or may be twice in an hour without using row by row insert rather using bulk.. somethng like "Insert into Tab1.. select * from Stg_tab1", so here the length of transaction will be shorter, and the Long running SELECT queries will be least affected by the UNDO generated from these INSERTS?

or by current row by row insert , if we will just increase the COMMIT duration 10k rows(one commit in each 10k rows), will that help in improving the UNDO reads? or by decreasing the concurrent batch inserts threads from java?

Actually i was testing how much UNDO gets generated by the INSERT(Conventional VS Bulk VS direct path) in presence of index in my transaction table(tab1). Below is the results, and it seems to me like in case of both BULK and Direct path insert the UNDO generated is lesser compared to the row by row inserts.initially when the table is blank i see major benefit with BULK/Direct path insert over row by row insert but when i tested with 10 million existing rows inside the table, the UNDO increased for Bulk/Direct path insert comparatively, while it was constant in case of row by row insert. And in our case as for each daily partion we consume ~150 million records, so during the insert after ~50 million records we may not see the benefit as we have 10 local indexes present in the table correct? So want your opinion if i am missing anything here? or we would really see this effect of lesser UNDO genaration if we will make the insert BULK(with lesser frequency i.e once or twice in an hour) rather row by row which is happening at current scenario 24/7.

**************************** Inserting into blank table
For conventional insert:- (10000 individual inserts in blank table)

Stats from v$transaction:-
USED_UREC USED_UBLK USED_UBLK(in KB)
100000 1310 10480

undo change vector size from v$mystat : 14250016


For bulk insert - (INSERT INTO Tab1 Select * from stage_tab where rownum<10001)

Stats from v$transaction:-
USED_UREC USED_UBLK USED_UBLK(in KB)
49788 824 6592

undo change vector size from v$mystat : 8340848

For Direct path Insert:- (INSERT /*+APPEND*/ INTO Tab1 Select * from stage_tab where rownum<10001)

Stats from v$transaction:-
USED_UREC USED_UBLK USED_UBLK(in KB)
743 299 2392

undo change vector size from v$mystat: 4672896

**************************************Insert into table having ~10million records in it

For conventional insert:- (10000 individual inserts in to same tabkle having ~10milion existing records)

Stats from v$transaction:-
USED_UREC USED_UBLK USED_UBLK(in KB)
100000 1338 10704 - 10 million rows present

undo change vector size from v$mystat : 12835048


For bulk insert - (INSERT INTO Tab1 Select * from stage_tab where rownum<10001)

Stats from v$transaction:-
USED_UREC USED_UBLK USED_UBLK(in KB)
28540 572 4576

undo change vector size from v$mystat : 8911496

For Direct path Insert:- (INSERT /*+APPEND*/ INTO Tab1 Select * from stage_tab where rownum<10001)

Stats from v$transaction:-
USED_UREC USED_UBLK USED_UBLK(in KB)
14836 477 3816

undo change vector size from v$mystat: 6065576

select "undo" on full table scan

GregMist, November 06, 2023 - 9:14 am UTC

Hi,
Thanks for your explanations, that's really interesting

I am having the same Issue but the undo block read does not append on index read but table block read while doing full table partition scan.
The other difference is that the insert is executed on an Oracle primary database and the select appends on the physical standby database
On the primary database, there is a continual insert on one table by many different processes at high rate.
On the standby database, there are extraction sql requests that are doing full partition table scan.

At some point, sessions are only doing “db file sequential read” on the undo tablespace and performances are really poor.

As I understood in your first reply, when doing full table scan oracle where able to easily discard the newly inserted rows while doing full table scan comparing at when doing index scan.
It has to be noticed that there are multiple sessions doing insert on the primary database with multiple commited transactions

How would you explain this sessions doing only undo sequential block read while doing full partition table scan?

Database version is 19.10

Here after the execution plan of the sql doing the full table partition scan on table T_1 on the standby database
--------------------------------------------------------------------------------------------------------
| Id  | Operation                      | Name  | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
--------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT               |       |       |       |    38 (100)|          |       |       |
|   1 |  SORT GROUP BY                 |       |     1 |    74 |    38   (6)| 00:00:01 |       |       |
|*  2 |   HASH JOIN                    |       |     1 |    74 |    37   (3)| 00:00:01 |       |       |
|*  3 |    HASH JOIN                   |       |     1 |    60 |    14   (8)| 00:00:01 |       |       |
|   4 |     VIEW                       |       |     1 |    61 |     3  (34)| 00:00:01 |       |       |
|   5 |      HASH GROUP BY             |       |     1 |    49 |     3  (34)| 00:00:01 |       |       |
|*  6 |       FILTER                   |       |       |       |            |          |       |       |
|   7 |        PARTITION RANGE ITERATOR|       |     1 |    49 |     2   (0)| 00:00:01 |   KEY |   KEY |
|   8 |         PARTITION LIST ALL     |       |     1 |    49 |     2   (0)| 00:00:01 |     1 |  LAST |
|*  9 |          TABLE ACCESS FULL     | T_1   |     1 |    49 |     2   (0)| 00:00:01 |   KEY |   KEY |
|  10 |     TABLE ACCESS FULL          | T_2   |  2207 | 33105 |    11   (0)| 00:00:01 |       |       |
|  11 |    TABLE ACCESS FULL           | T_3   |  6543 |   191K|    23   (0)| 00:00:01 |       |       |
--------------------------------------------------------------------------------------------------------


And here, extract of the activity of the session exécuting the request

SQL> select sid, event, case p1text when 'file#' then FILE_NAME else to_char(p1) end wait_parameter, wait_TIME_MICRO/1000 WAIT_MS 
  2  from V$SESSION_WAIT_HISTORY W
  3  LEFT OUTER JOIN dba_data_files F ON F.file_id=w.P1
  4  where sid=726;

       SID EVENT                     WAIT_PARAMETER                                 WAIT_MS
---------- ------------------------- ---------------------------------------------- ----------
       726 db file sequential read   +DATA01/DB1/DATAFILE/undotbs1.1351.1087548745  ,426      
       726 db file sequential read   +DATA01/DB1/DATAFILE/undotbs1.1351.1087548745  ,434      
       726 db file sequential read   +DATA01/DB1/DATAFILE/undotbs1.1351.1087548745  ,264      
       726 db file sequential read   +DATA01/DB1/DATAFILE/undotbs1.1351.1087548745  1,907
       726 db file sequential read   +DATA01/DB1/DATAFILE/undotbs1.1351.1087548745  ,434      
       726 db file sequential read   +DATA01/DB1/DATAFILE/undotbs1.1351.1087548745  ,514      
       726 db file sequential read   +DATA01/DB1/DATAFILE/undotbs1.1351.1087548745  ,524      
       726 db file sequential read   +DATA01/DB1/DATAFILE/undotbs1.1351.1087548745  2,666
       726 db file sequential read   +DATA01/DB1/DATAFILE/undotbs1.1351.1087548745  ,432      
       726 db file sequential read   +DATA01/DB1/DATAFILE/undotbs1.1351.1087548745  ,736      


Thanks for your help.
Connor McDonald
November 07, 2023 - 5:43 am UTC

Here's a hypothesis to explore:

The database is "lazy", ie, we'll try not to do work unless we have to. So when you do a lot of DML into a database, we often won't go to all of the effort of making sure all the header information etc in the block is kept perfectly up to date. After all, that's just extra work that we really only need to do if someone comes along later and wants to query the data. Hence the term "delayed block cleanout" - the *next* person who comes along to query that data pays the price of cleaning up the mess left by the person who did all the changes. Once the *second* person cleans up the mess, the block is now pristine and no-one has to pay that price again.

But consider an Active Dataguard setup. Someone on the *primary* does lots of DML, leaving a big mess in the block.

Now the "next" people reading that data are not doing so on the primary, they are doing so on the Standby. They have to clean up the mess and thus they read the undo information to make sure the changes in the blocks are committed/ready to use...

BUT ... its a read-only system. They can't *clean* the blocks, so *every* person running a query is going to pay that price in some way.

Something to try - run something like this on your primary:

alter session set "_serial_direct_read" = never;
select max(unindexed_column) from your_table

which will clean the blocks on the primary, which will make its way down to the standby

and see if that reduces/eliminates all the undo work on the standby