Skip to Main Content
  • Questions
  • Surprise findings on Index Fast Full Scans using parallel when table set to not parallel

Breadcrumb

Question and Answer

Tom Kyte

Thanks for the question, Matthew.

Asked: March 10, 2008 - 11:20 pm UTC

Last updated: April 29, 2008 - 6:33 pm UTC

Version: 10.1.2

Viewed 1000+ times

You Asked

I was a bit surprised to notice in my development environment a couple of months, a large number of sessions that were invoking parallel operations, especially since I made sure to set all tables in the schema to no parallel. I brushed it off as something I needed to look into when I had the time, because I couldn't find the cause at first glance.

Then two weeks ago I was tasked with determining in a straight forward testable manner for a client the best indexing strategy that should be used in an ODS or data wharehouse, as they had been informed by other consultants that because Oracle can now perform index fast full scans, range scans and skip scans of composite indexes, that all they really needed were b-tree multi column indexes on their tables. So I cracked out your book Expert Oracle Database Architecture, built a big_table, and populated it from all_object multiple times, using your standard script, and proceded to crank out autotraces on common queries. Then I noticed that even though my table was set to noparallel, I was getting parallel operations in my explain plans.

SQL> set autotrace traceonly
SQL> select owner from big_table
2 where owner =USER;

312 rows selected.


Execution Plan
----------------------------------------------------------
Plan hash value: 3993303771

-------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 28233 | 165K| 1279 (1)| 00:00:16 |
|* 1 | TABLE ACCESS FULL| BIG_TABLE | 28233 | 165K| 1279 (1)| 00:00:16 |
-------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

1 - filter("OWNER"=USER@!)


Statistics
----------------------------------------------------------
247 recursive calls
0 db block gets
5789 consistent gets
5692 physical reads
376 redo size
3631 bytes sent via SQL*Net to client
570 bytes received via SQL*Net from client
22 SQL*Net roundtrips to/from client
5 sorts (memory)
0 sorts (disk)
312 rows processed

SQL> create unique index big_tab_comp_uix on big_table
2 (owner, object_type,object_name,created,id)
3 nologging
4 parallel (degree default instances default);

Index created.

SQL> select owner, object_name
2 from big_table
3 where owner=USER;

312 rows selected.

Execution Plan
----------------------------------------------------------
Plan hash value: 1440650364
-------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 28233 | 854K| 246 (1)| 00:00:03 |
|* 1 | INDEX RANGE SCAN| BIG_TAB_COMP_UIX | 28233 | 854K| 246 (1)| 00:00:03 |
-------------------------------------------------------------------------------------


Predicate Information (identified by operation id):
---------------------------------------------------

1 - access("OWNER"=USER@!)


Statistics
----------------------------------------------------------
1 recursive calls
0 db block gets
26 consistent gets
4 physical reads
0 redo size
4414 bytes sent via SQL*Net to client
570 bytes received via SQL*Net from client
22 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
312 rows processed

SQL> select owner, object_name
2 from big_table
3 where object_type ='TABLE';

13768 rows selected.

Execution Plan
----------------------------------------------------------
Plan hash value: 2757377107

-------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | TQ |IN-OUT| PQ Distrib |
-------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 13062 | 510K| 752 (2)| 00:00:10 | | | |
| 1 | PX COORDINATOR | | | || | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10000 | 13062 | 510K| 752 (2)| 00:00:10 | Q1,00 | P->S | QC (RAND) |
| 3 | PX BLOCK ITERATOR | | 13062 | 510K| 752 (2)| 00:00:10 | Q1,00 | PCWC | |
|* 4 | INDEX FAST FULL SCAN| BIG_TAB_COMP_UIX | 13062 | 510K| 752 (2)| 00:00:10 | Q1,00 | PCWP | |
-------------------------------------------------------------------------------------------------------------------------


Predicate Information (identified by operation id):
---------------------------------------------------

4 - filter("OBJECT_TYPE"='TABLE')


Statistics
----------------------------------------------------------
45 recursive calls
3 db block gets
4155 consistent gets
3423 physical reads
636 redo size
182888 bytes sent via SQL*Net to client
10437 bytes received via SQL*Net from client
919 SQL*Net roundtrips to/from client
1 sorts (memory)
0 sorts (disk)
13768 rows processed


SQL> alter index big_tab_comp_uix noparallel;

Index altered.

SQL> select owner, object_name
2 from big_table
3 where object_type ='TABLE';

13768 rows selected.


Execution Plan
----------------------------------------------------------
Plan hash value: 4265662311

-----------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-----------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 13062 | 510K| 752 (2)| 00:00:10 |
|* 1 | INDEX FAST FULL SCAN| BIG_TAB_COMP_UIX | 13062 | 510K| 752 (2)| 00:00:10 |
-----------------------------------------------------------------------------------------


Predicate Information (identified by operation id):
---------------------------------------------------

1 - filter("OBJECT_TYPE"='TABLE')


Statistics
----------------------------------------------------------
262 recursive calls
0 db block gets
4423 consistent gets
3414 physical reads
0 redo size
179036 bytes sent via SQL*Net to client
10437 bytes received via SQL*Net from client
919 SQL*Net roundtrips to/from client
5 sorts (memory)
0 sorts (disk)
13768 rows processed

SQL>

So I'm guessing that because Oracle treats the index as a skinny table on an Index Fast Full Scan it uses the degree of parallelzation of the index when executing the query. This is not necessarily a bad thing except for thsoe of use who hate waiting on index builds (like me) who builds all indexs to the defautl degree of parallel in my code, I now have to remember to alter the index after the fact or have multiple sessions invoking parallel operations when they shouldn't.

When are you going to be publishing an update to your other book? I can't wait to get more tips, tidbits and actual real world examples to use in my code,

Thanks

and Tom said...

... as they had been informed by other consultants that because Oracle can now perform index fast full scans, range scans and skip scans of composite indexes, that all they really needed were b-tree multi column indexes on their tables. ....

wow, is that just about the wrongest statement ever uttered? It ranks up there, way way up there. No, that is absolutely not even close to being remotely true. bitmap indexes play a big role, huge role.


you created your indexes with parallel - they are parallel enabled in addition to being built in parallel.

All of your indexes would be subject to parallel query therefore - you would want to "un-parallel" them if this is not what you desire.

... This is not necessarily a bad thing except for thsoe of use who hate waiting on index builds (like me) who builds all indexs to the defautl degree of parallel in my code, I now have to remember to alter the index after the fact or have multiple sessions invoking parallel operations when they shouldn't. ....

that was always true however, has been true.. It is true of TABLES as well...

ops$tkyte%ORA10GR2> create table t parallel 4 as select * from big_table.big_table;

Table created.

ops$tkyte%ORA10GR2> @at
ops$tkyte%ORA10GR2> select count(*) from t;

Execution Plan
----------------------------------------------------------
Plan hash value: 3126468333

------------------------------------------------------------------------
| Id  | Operation              | Name     | Rows  | Cost (%CPU)| Time
------------------------------------------------------------------------
|   0 | SELECT STATEMENT       |          |     1 |    94   (2)| 00:00:0
|   1 |  SORT AGGREGATE        |          |     1 |            |
|   2 |   PX COORDINATOR       |          |       |            |
|   3 |    PX SEND QC (RANDOM) | :TQ10000 |     1 |            |
|   4 |     SORT AGGREGATE     |          |     1 |            |
|   5 |      PX BLOCK ITERATOR |          |   100K|    94   (2)| 00:00:0
|   6 |       TABLE ACCESS FULL| T        |   100K|    94   (2)| 00:00:0
------------------------------------------------------------------------

Note



if you create parallel, you've created parallel....

Rating

  (7 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

Thanks

Matthew, March 11, 2008 - 10:31 pm UTC

wow, is that just about the wrongest statement ever uttered? It ranks up there, way way up there. No, that is absolutely not even close to being remotely true. bitmap indexes play a big role, huge role.

I thought so as well, which is why I spent an entire week setting up the tables, building a star schema out of the data in big table and then systematicaly, documenting step by step how the different types of indexes worked in that set up.

It was also nice having Jonathan Lewis's article on bitmap indexes, bitmap star transformations and bitmap join indexes as a referenece, I even quoted him twice on the caveats of bitmaps indexes, bitmap join indexes and ensuring that tables stats are current and available for the CBO to work with.

you created your indexes with parallel - they are parallel enabled in addition to being built in parallel.

I was just a bit surprised since when the index was being ranged scanned I didn't see the parallel kick in. I really didn't notice the parallel kick in until there was an INDEX FAST FULL SCAN


if you create parallel, you've created parallel....

After reading and absorbing a ton of great information from this site your book, and others like it, I have realized that before I was using parallel as a crutch to help me brute force through some non too optimal SQL. Now, it is used only if I'm dealing with huge (1 million+) set of data (or DDL), and then only after I've gotten the sql to run as fast as I can as part of a normal SQL process. I'd like to make it so that my data loads take only a little bit longer as it normally takes to read the data out of the table. Once I get to that point then if I really need more speed I can put in a parallel hint.

Thanks again for all the great information


Q. for Matthew

BrianB, March 12, 2008 - 3:19 pm UTC

Out of curiosity, why the aversion to parallel operations?

I'm intrigued by the reasoning - I expect there's something to learn from your viewpoint.
Tom Kyte
March 12, 2008 - 5:53 pm UTC

when parallel query first came out, 7.1.6 was the production release of Oracle.

A data warehouse was a new concept.

They typically had about one user running queries. So, for that one user, using parallel query made sense.

Once you have many more users than CPU's/IO channels - parallel stops making perfect sense. It is very good for administrative tasks, but todays "warehouse" with 16 cpu's and 100 concurrent users - parallel is many times not reasonable (my preference is to ask for it via a parallel hint without a degree - if sufficient resources exist, it'll do it - else it won't)

<quote src=expert oracle database architecture>

When to Use Parallel Execution

Parallel execution can be fantastic. It can allow you to take a process that executes over many hours or days and complete it in minutes. Breaking down a huge problem into small components may, in some cases, dramatically reduce the processing time. However, one underlying concept that it will be useful to keep in mind while considering parallel execution is summarized by this very short quote from Practical Oracle8i: Building Efficient Databases (Addison-Wesley, 2001) by Jonathan Lewis:

PARALLEL QUERY option is essentially nonscalable.

Parallel execution is essentially a nonscalable solution. It was designed to allow an individual user or a particular SQL statement to consume all resources of a database. If you have a feature that allows an individual to make use of everything that is available, and then allow two individuals to use that feature, you¿ll have obvious contention issues. As the number of concurrent users on your system begins to overwhelm the number of resources you have (memory, CPU, and I/O), the ability to deploy parallel operations becomes questionable. If you have a four-CPU machine, for example, and on average you have 32 users executing queries simultaneously, then the odds are that you do not want to parallelize their operations. If you allowed each user to perform just a ¿parallel 2¿ query, then you would now have 64 concurrent operations taking place on a machine with just four CPUs. If the machine were not overwhelmed before parallel execution, it almost certainly would be now.
In short, parallel execution can also be a terrible idea. In many cases, the application of parallel processing will only lead to increased resource consumption, as parallel execution attempts to use all available resources. In a system where resources must be shared by many concurrent transactions, such as an OLTP system, you would likely observe increased response times due to this. It avoids certain execution techniques that it can use efficiently in a serial execution plan and adopts execution paths such as full scans in the hope that by performing many pieces of the larger, bulk operation in parallel, it would be better than the serial plan. Parallel execution, when applied inappropriately, may be the cause of your performance problem, not the solution for it.

So, before applying parallel execution, you need the following two things to be true:
 * You must have a very large task, such as the full scan of 50GB of data.
 * You must have sufficient available resources. Before parallel full scanning 50GB of data, you would want to make sure that there is sufficient free CPU (to accommodate the parallel processes) as well as sufficient I/O. The 50GB should be spread over more than one physical disk to allow for many concurrent read requests to happen simultaneously, there should be sufficient I/O channels from the disk to the computer to retrieve the data from disk in parallel, and so on.

If you have a small task, as generally typified by the queries carried out in an OLTP system, or you have insufficient available resources, again as is typical in an OLTP system where CPU and I/O resources are often already used to their maximum, then parallel execution is not something you¿ll want to consider. So you can better understand this concept, I present the following analogy.

Answer for Brain B

Matthew, March 12, 2008 - 7:41 pm UTC

Tom nailed it as usual.

I also have an example for why I try an avoid even the parallel hints.

My ETL is currently being run as a series of packages with one driver calling multiple other packages. (This is cause of our security posture which state I can't run DDL in a schema unless I'm the schema owner). This system is also receiving CDC and Streams replication, and during the same time frame that the ETL is running the DBA's felt that it was essentil to run RMAN as well as perform the DBMS_STATS on the database schema's. All of these "System" processes run in parallel and unfortunately are controlled by separate DBMS_JOBs, which means I currently have no way of telling when they are running, or when they are set to run as I do not have visability into the System schema in my production system (yeap another security thing).

So I avoid the potential parallel server contention issue by controlling the one thing I can in this situation... by not using parallel unless I absolutely have to.

Another benefit of this is that it makes me write better SQL. Like I said before, when I started out, I was using parallel as a brute force approach to running the ETL. I write much better code now, and can now run the same process that I was using parallel with my new code in the same amount of time without using parallel.

Thanks

Makes Sense

BrianB, March 13, 2008 - 9:37 am UTC

Tom and Matthew, thanks!

That all makes sense. I hadn't considered the scalability problem it adds.

I learn something new here every day.

Parallel INDEX RANGE SCAN ?

Nico, April 22, 2008 - 9:33 am UTC

Quoting Matthew :
"I was just a bit surprised since when the index was being ranged scanned I didn't see the parallel kick in. I really didn't notice the parallel kick in until there was an INDEX FAST FULL SCAN"

Can an INDEX RANGE SCAN be done in parallel ? (on a non-partioned B-tree index of a non-partitioned heap organized table...)

I can imagine it to be possible : Splitting the range in pieces, each parallel process "rangescanning" it's own piece... but I might be missing some spécial cases that would make this not correct.
(beside that it might not be the most efficient way to get the results)

Wish you could shed some light on that.
(Ignore it of course, if it's some kind of new question.)
Tom Kyte
April 23, 2008 - 5:57 pm UTC

well, we can do many index range scans in parallel (for a single query), but each single range scan would be done serially...

Degree of parallelism

A reader, April 24, 2008 - 8:47 am UTC

Tom,
If I limit the degree of parallelism to say 4, then, given enough resources, I can still use parallel query. Is this correct?

Thanks...
Tom Kyte
April 28, 2008 - 11:40 am UTC

you can always use parallel query.

whether it helps you or hurts you is another question entirely (and one that we have insufficient data to even begin to comment on here)

Degree of parallelism

A reader, April 29, 2008 - 12:35 pm UTC

Apologize. I got carried away by the thread and did not ask the question correctly.

Is there a way to measure the impact of parallel query on IO?
Thanks

Tom Kyte
April 29, 2008 - 6:33 pm UTC

define "impact"

you can run a query with and without parallel and measure the number of IO's (logical and physical) - and see the difference - if that is what you mean by "impact"


A parallel query will tend to use direct IO - bypassing the cache, a non parallel one typically will not do that - so they will almost certainly "be different" in that respect