Skip to Main Content
  • Questions
  • "Filter" step during chunk based processing.

Breadcrumb

Question and Answer

Connor McDonald

Thanks for the question, Rajeshwaran.

Asked: November 05, 2020 - 10:28 am UTC

Last updated: November 12, 2020 - 2:04 am UTC

Version: 18.10

Viewed 1000+ times

You Asked

Team:

Please find below the test case(modelled like our business requirements) along with execution plan (to show predicates section)
and the sql-monitor report for timings, this was run on Oracle 18c from Exacc platform.

Please help us to understand why the optimizer generated the "filter" at step#2 in plan?
In our real execution we could see - that particular step took around, 12min to 15 min (result set is getting buffered and that filter is getting validated)
so how can we get rid of that "filter"?

drop table driver purge;
drop table t1 purge;
drop table t2 purge;
drop table t3 purge;

create table driver as select a.* from all_objects a,  
 (select rownum from dual connect by level <=10 ) ;
create table t1 as select a.* from all_objects a,  
 (select rownum from dual connect by level <=100 );
create table t2 as select t1.* from t1;
create table t3( object_id number, data_object_id number );


exec dbms_parallel_execute.create_task(task_name=>'DEMO_TASK');
begin 
 dbms_parallel_execute.create_chunks_by_rowid(
  task_name=>'DEMO_TASK',
  table_owner=>user,
  table_name=>'DRIVER',
  by_row=>false,
  chunk_size=>100);
end;
/
select count(*) from user_parallel_execute_chunks where task_name='DEMO_TASK';

declare
 l_sql long;
begin
 l_sql :=' insert into t3( object_id, data_object_id) 
  select t1.object_id, t2.data_object_id
  from t1, t2, driver t3 
  where t1.object_id = t2.object_id
  and t2.object_id = t3.object_id
  and t3.rowid between :start_id and :end_id ';
 dbms_parallel_execute.run_task(
  task_name=>'DEMO_TASK',
  sql_stmt=>l_sql,
  language_flag=>dbms_sql.native,
  parallel_level=>4);
end;
/ 

demo@PDB1> select * from table( dbms_xplan.display_cursor('9cbx808m23rkg'));

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
SQL_ID  9cbx808m23rkg, child number 0
-------------------------------------
 insert into t3( object_id, data_object_id) select t1.object_id,
t2.data_object_id from t1, t2, driver t3 where t1.object_id =
t2.object_id and t2.object_id = t3.object_id and t3.rowid between
:start_id and :end_id

Plan hash value: 3603417986

----------------------------------------------------------------------------------------------------------
| Id  | Operation                               | Name   | Rows  | Bytes |TempSpc| Cost (%CPU)| Time     |
----------------------------------------------------------------------------------------------------------
|   0 | INSERT STATEMENT                        |        |       |       |       |   303K(100)|          |
|   1 |  LOAD TABLE CONVENTIONAL                | T3     |       |       |       |            |          |
|*  2 |   FILTER                                |        |       |       |       |            |          |
|*  3 |    HASH JOIN                            |        |    61M|  1825M|    22M|   303K  (1)| 00:00:12 |
|*  4 |     HASH JOIN                           |        |   622K|    15M|       |   148K  (1)| 00:00:06 |
|*  5 |      TABLE ACCESS STORAGE BY ROWID RANGE| DRIVER |  6267 |   104K|       | 13586   (1)| 00:00:01 |
|   6 |      TABLE ACCESS STORAGE FULL          | T2     |    25M|   215M|       |   135K  (1)| 00:00:06 |
|   7 |     TABLE ACCESS STORAGE FULL           | T1     |    25M|   119M|       |   135K  (1)| 00:00:06 |
----------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - filter(CHARTOROWID(:END_ID)>=CHARTOROWID(:START_ID))
   3 - access("T1"."OBJECT_ID"="T2"."OBJECT_ID")
   4 - access("T2"."OBJECT_ID"="T3"."OBJECT_ID")
   5 - storage("T3".ROWID>=CHARTOROWID(:START_ID) AND "T3".ROWID<=CHARTOROWID(:END_ID))


30 rows selected.

SQL Monitoring Report

SQL Text
------------------------------
insert into t3( object_id, data_object_id) select t1.object_id, t2.data_object_id from t1, t2, driver t3 where t1.object_id = t2.object_id and t2.object_id = t3.object_id and t3.rowid between :start_id and :end_id

Global Information
------------------------------
 Status              :  DONE                  
 Instance ID         :  4                     
 Session             :  DEMO (196:63770) 
 SQL ID              :  9cbx808m23rkg         
 SQL Execution ID    :  67108882              
 Execution Started   :  11/05/2020 04:45:43   
 First Refresh Time  :  11/05/2020 04:45:46   
 Last Refresh Time   :  11/05/2020 04:46:14   
 Duration            :  31s                   
 Module/Action       :  SQL*Plus/-            
 Service             :  pdb1                 
 Program             :  sqlplus.exe           

Binds
========================================================================================================================
|   Name    | Position |     Type     |                                     Value                                      |
========================================================================================================================
| :START_ID |        1 | VARCHAR2(32) | AABRIxAADAADAJ0AAA                                                             |
| :END_ID   |        2 | VARCHAR2(32) | AABRIxAADAADALXH//                                                             |
========================================================================================================================

Global Stats
===========================================================================================================================
| Elapsed |   Cpu   |    IO    |  Other   | Buffer | Read  | Read  | Uncompressed |  Offload   |    Offload     |  Cell   |
| Time(s) | Time(s) | Waits(s) | Waits(s) |  Gets  | Reqs  | Bytes |    Bytes     | Elig Bytes | Returned Bytes | Offload |
===========================================================================================================================
|      31 |      23 |     7.70 |     0.59 |     2M | 25547 |   8GB |        566MB |        8GB |           30MB |  99.62% |
===========================================================================================================================

SQL Plan Monitoring Details (Plan Hash Value=3603417986)
==============================================================================================================================================================================================
| Id |                 Operation                 |  Name  |  Rows   | Cost  |   Time    | Start  | Execs |   Rows   | Read  | Read  |  Mem  | Activity |           Activity Detail           |
|    |                                           |        | (Estim) |       | Active(s) | Active |       | (Actual) | Reqs  | Bytes | (Max) |   (%)    |             (# samples)             |
==============================================================================================================================================================================================
|  0 | INSERT STATEMENT                          |        |         |       |        29 |     +3 |     1 |        0 |       |       |     . |          |                                     |
|  1 |   LOAD TABLE CONVENTIONAL                 | T3     |         |       |        30 |     +2 |     1 |        0 | 17659 | 138MB |     . |    77.42 | log buffer space (1)                |
|    |                                           |        |         |       |           |        |       |          |       |       |       |          | Cpu (14)                            |
|    |                                           |        |         |       |           |        |       |          |       |       |       |          | cell single block physical read (9) |
|  2 |    FILTER                                 |        |         |       |        29 |     +3 |     1 |      55M |       |       |     . |          |                                     |
|  3 |     HASH JOIN                             |        |     62M |  303K |        29 |     +3 |     1 |      55M |       |       |  49MB |    19.35 | Cpu (6)                             |
|  4 |      HASH JOIN                            |        |    622K |  149K |         3 |     +1 |     1 |     550K |       |       |   6MB |     3.23 | Cpu (1)                             |
|  5 |       TABLE ACCESS STORAGE BY ROWID RANGE | DRIVER |    6267 | 13586 |         1 |     +3 |     1 |     5504 |     2 | 784KB |   1MB |          |                                     |
|  6 |       TABLE ACCESS STORAGE FULL           | T2     |     25M |  135K |         1 |     +3 |     1 |      25M |  3898 |   4GB |   6MB |          |                                     |
|  7 |      TABLE ACCESS STORAGE FULL            | T1     |     25M |  135K |        29 |     +3 |     1 |      25M |  3898 |   4GB |   6MB |          |                                     |
==============================================================================================================================================================================================

and Connor said...

Can we see some SQL monitoring info on the *real* query, because with your example, the FILTER time looks like its negligible delay at all, and I'm seeing the same when I repeat your demo here.

SQL> create table driver as select a.* from dba_objects a,
  2   (select rownum from dual connect by level <=10 ) ;

Table created.

SQL> create table t1 as select a.* from dba_objects a,
  2   (select rownum from dual connect by level <=100 );

Table created.

SQL> create table t2 as select t1.* from t1;

Table created.

SQL> create table t3( object_id number, data_object_id number );

Table created.

SQL> variable start_id varchar2(100)
SQL> variable END_ID varchar2(100)
SQL> exec :start_id := 'AAARfBAAAAAAAOLAAA'; :end_id := 'AAARfBAAAAAABSZAIE';

PL/SQL procedure successfully completed.

SQL>
SQL>
SQL>    select /*+ gather_plan_statistics */ max(t1.object_id), max(t2.data_object_id)
  2    from t1, t2, driver t3
  3    where t1.object_id = t2.object_id
  4    and t2.object_id = t3.object_id
  5    and t3.rowid >= chartorowid(:start_id) and t3.rowid<=chartorowid(:end_id)
  6  /

MAX(T1.OBJECT_ID) MAX(T2.DATA_OBJECT_ID)
----------------- ----------------------
            71612                  71602

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                                  | Name                       | Starts | E-Rows | A-Rows |   A-Time   | Buffers | Reads  |  OMem |  1Mem | Used-Mem |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                           |                            |      1 |        |      1 |00:00:54.93 |   11106 |  10311 |       |       |          |
|   1 |  RESULT CACHE                              | 3910hufqshac7cv5ahmutw6kt1 |      1 |      1 |      1 |00:00:54.93 |   11106 |  10311 | 65536 |  1024 |          |
|   2 |   SORT AGGREGATE                           |                            |      1 |      1 |      1 |00:00:54.93 |   11106 |  10311 |       |       |          |
|*  3 |    FILTER                                  |                            |      1 |        |    517M|00:00:50.03 |   11106 |  10311 |       |       |          |
|*  4 |     HASH JOIN                              |                            |      1 |     12M|    517M|00:00:47.83 |   11106 |  10311 |   255M|    23M|  292M (0)|
|   5 |      JOIN FILTER CREATE                    | :BF0000                    |      1 |    126K|   5177K|00:00:00.49 |    5940 |   5148 |       |       |          |
|*  6 |       HASH JOIN RIGHT SEMI                 |                            |      1 |    126K|   5177K|00:00:00.36 |    5940 |   5148 |    27M|  6846K|   24M (0)|
|   7 |        JOIN FILTER CREATE                  | :BF0001                    |      1 |   1294 |    516K|00:00:00.02 |     789 |      0 |       |       |          |
|*  8 |         TABLE ACCESS STORAGE BY ROWID RANGE| DRIVER                     |      1 |   1294 |    516K|00:00:00.01 |     789 |      0 |  1025K|  1025K|          |
|   9 |        JOIN FILTER USE                     | :BF0001                    |      1 |   5177K|   5177K|00:00:00.23 |    5151 |   5148 |       |       |          |
|* 10 |         TABLE ACCESS STORAGE FULL          | T2                         |      1 |   5177K|   5177K|00:00:00.22 |    5151 |   5148 |  1025K|  1025K| 8227K (0)|
|  11 |      JOIN FILTER USE                       | :BF0000                    |      1 |   5177K|   5177K|00:00:00.12 |    5166 |   5163 |       |       |          |
|* 12 |       TABLE ACCESS STORAGE FULL            | T1                         |      1 |   5177K|   5177K|00:00:00.10 |    5166 |   5163 |  1025K|  1025K| 9256K (0)|
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------


So I got a 3 second overhead for 500 million rows.

Rating

  (1 rating)

Is this answer out of date? If it is, please let us know via a Comment

Comments

few more details.

Rajeshwaran, jeyabal, November 11, 2020 - 7:03 am UTC

Here is from the application database.
the join at step#3 - started at 612 sec and active till 3588 sec - so the total time consumed by this step is (3588-162 = 2679 sec) - is that correct?
if so the step#2 - filter - took around 2388 seconds of this entire execution?

Global Stats
============================================================================================================================================================
| Elapsed |   Cpu   |    IO    | Application | Concurrency | Cluster  | Buffer | Read | Read  | Write | Write | Uncompressed |  Offload   |    Offload     |
| Time(s) | Time(s) | Waits(s) |  Waits(s)   |  Waits(s)   | Waits(s) |  Gets  | Reqs | Bytes | Reqs  | Bytes |    Bytes     | Elig Bytes | Returned Bytes |
============================================================================================================================================================
|    4216 |    2082 |     2126 |        0.02 |        2.83 |     4.76 |    19M | 708K | 130GB |  472K |  11GB |        128GB |      119GB |          167GB |
============================================================================================================================================================

SQL Plan Monitoring Details (Plan Hash Value=268960802)
=======================================================================================================================================================================================================================
| Id |                   Operation                    |  Name  |  Rows   | Cost |   Time    | Start  | Execs |   Rows   | Read  | Read  | Write | Write |  Mem  | Temp  | Activity | Activity Detail |
|    |                                                |                         | (Estim) |      | Active(s) | Active |       | (Actual) | Reqs  | Bytes | Reqs  | Bytes | (Max) | (Max) |   (%)    |   (# samples)   |
=======================================================================================================================================================================================================================
|  0 | INSERT STATEMENT                               |         |         |      |      3592 |   +608 |     1 |        0 |       |       |       |       |     . |     . |          |                 |
|  1 |   LOAD TABLE CONVENTIONAL                      | T8      |         |      |      3295 |   +905 |     1 |        0 |  480K |   4GB |  457K |   4GB |     . |     . |          |                 |
|  2 |    FILTER                                      |         |         |      |      3294 |   +906 |     1 |     469K |       |       |       |       |     . |     . |          |                 |
|  3 |     HASH JOIN OUTER                            |         |   99185 |  11M |      3588 |   +612 |     1 |     469K | 10320 |   2GB | 10320 |   2GB | 410MB |   3GB |          |                 |
|  4 |      HASH JOIN OUTER                           |         |   99185 |   8M |       427 |   +414 |     1 |     469K |  2288 |   2GB |  2288 |   2GB | 412MB |   2GB |          |                 |
|  5 |       HASH JOIN OUTER                          |         |   99185 |   5M |       317 |   +294 |     1 |     469K |  1485 |   1GB |  1485 |   1GB | 437MB |   1GB |          |                 |
|  6 |        HASH JOIN OUTER                         |         |   99185 |   4M |       195 |   +214 |     1 |     469K |  1193 |   1GB |  1193 |   1GB | 337MB |   1GB |          |                 |
|  7 |         HASH JOIN OUTER                        |         |   99185 |   3M |       289 |     +4 |     1 |     469K |       |       |       |       |  59MB |     . |          |                 |
|  8 |          MERGE JOIN CARTESIAN                  |         |   99185 | 1834 |         1 |     +4 |     1 |     469K |       |       |       |       |     . |     . |          |                 |
|  9 |           TABLE ACCESS STORAGE FULL            | T1      |       1 |    6 |         1 |     +4 |     1 |        1 |       |       |       |       |     . |     . |          |                 |
| 10 |           BUFFER SORT                          |         |   99185 | 1828 |         1 |     +4 |     1 |     469K |       |       |       |       |  32MB |     . |          |                 |
| 11 |            TABLE ACCESS STORAGE BY ROWID RANGE | T2      |   99185 | 1828 |         1 |     +4 |     1 |     469K |     2 | 384KB |       |       |   2MB |     . |          |                 |
| 12 |          TABLE ACCESS STORAGE FULL             | T3    |     40M |   2M |       292 |     +1 |     1 |      40M |  111K |  69GB |       |       |   3MB |     . |          |                 |
| 13 |         TABLE ACCESS STORAGE FULL              | T4      |     40M | 124K |        95 |   +292 |     1 |      40M | 12929 |   3GB |       |       |   3MB |     . |          |                 |
| 14 |        TABLE ACCESS STORAGE FULL               | T5      |     40M | 189K |       179 |   +408 |     1 |      40M | 18562 |   5GB |       |       |   3MB |     . |          |                 |
| 15 |       TABLE ACCESS STORAGE FULL                | T6   |     40M | 922K |       197 |   +610 |     1 |      40M | 44444 |  26GB |       |       |   3MB |     . |          |                 |
| 16 |      TABLE ACCESS STORAGE FULL                 | T7      |     40M | 545K |       573 |   +842 |     1 |      40M | 26737 |  15GB |       |       |   3MB |     . |          |                 |
=======================================================================================================================================================================================================================

Connor McDonald
November 12, 2020 - 2:04 am UTC

I don't think we can interpret it that way when it comes to nested hash joins because unlike (say) a merge join, rows will start coming out of the hash join as soon as we've completed the first half of the join and commenced on the second.

So whilst step3 starts at 612 and is active until 3588, rows can be produced from this join *throughout* this time, rather than just at the end. The +906 suggests the first rows started coming out at then and went through the filter. Its the duration of the underlying join that makes the filter look like its expensive.

An easy way to check this is

- create a table
- find min and max rowid via dba_extents
- compare query times for

select ... from table
select ... from table where rowid between min_rowid and max_rowid

I did that with a 2billion row table

SQL> set timing on
SQL> select max(y), count(*) cnt
  2  from tx
  3  where rowid >= chartorowid(:start_id) and rowid<=chartorowid(:end_id);

    MAX(Y)          CNT
---------- ------------
    100000   2000000000

Elapsed: 00:00:53.90
SQL>
SQL> select max(y), count(*) cnt
  2  from tx
  3  where rowid >= chartorowid(:start_id) and rowid<=chartorowid(:end_id);

    MAX(Y)          CNT
---------- ------------
    100000   2000000000

Elapsed: 00:00:54.29
SQL>
SQL> select max(y), count(*) cnt
  2  from tx;

    MAX(Y)          CNT
---------- ------------
    100000   2000000000

Elapsed: 00:00:53.95
SQL>
SQL> select max(y), count(*) cnt
  2  from tx;

    MAX(Y)          CNT
---------- ------------
    100000   2000000000

Elapsed: 00:00:53.86
SQL>
SQL>



More to Explore

PL/SQL demos

Check out more PL/SQL tutorials on our LiveSQL tool.

PL/SQL docs

PL/SQL reference manual from the Oracle documentation library