Skip to Main Content
  • Questions
  • oracle is doing full table scans on a table with raw column and the column is primary key

Breadcrumb

Question and Answer

Tom Kyte

Thanks for the question.

Asked: October 28, 2008 - 8:58 pm UTC

Last updated: March 18, 2011 - 6:43 pm UTC

Version: 11.1.0.6

Viewed 10K+ times! This question is

You Asked

Hi Tom,
Thanks for your continuous support.
We have a table with blob data type as one column. We are using secure file option as we are using 11g. We kept the blob column in a seperate tablespace. In the main table we have a column called streamID which is RAW datatype. And this column is primary key index. So, when we are trying to retrieve the blob data using streamid, it is always doing full table scan. But, when I tried to use HEXTORAW then it is doing index scans. These queries are generated by application people using SQLGEN and Hibernate. We use bind variables too. I don't know whether hextoraw is the proper solution and if yes don't know how to implement it in the application.

*) sql query:

SELECT streamId, originalOffset, originalSize, chunkData FROM PackageStore where streamId= :"SYS_B_0";

explain plan:
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------

-----------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost |
-----------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 131 | 1749 |
| 1 | TABLE ACCESS FULL| PACKAGESTORESTREAMCHUNK | 1 | 131 | 1749 |
-----------------------------------------------------------------------------


*) Using Hextoraw function

SQL> explain plan for SELECT streamId, originalOffset,chunkData FROM PackageStore where streamId=hextoraw(:"SYS_B_0");

Explained.

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
----------

| Id | Operation | Name | Rows | Bytes
| Cost |

--------------------------------------------------------------------------------
----------

| 0 | SELECT STATEMENT | | 1 | 131

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
| 3 |

| 1 | TABLE ACCESS BY INDEX ROWID| PACKAGESTORESTREAMCHUNK | 1 | 131
| 3 |

| 2 | INDEX UNIQUE SCAN | PK_PACKAGESTORESTREAMCHUNK | 1 |
| 2 |

--------------------------------------------------------------------------------

SQL> select count(*) from packagestore;

COUNT(*)
----------
584768


and Tom said...

... We use bind variables too....


NO YOU DO NOT, YOU ARE USING CURSOR_SHARING=force or similar.

I can tell by this:

streamId= :"SYS_B_0";


If you go back to the application, you'll find it has something like:

streamId = '0123456789ABCDEF';

in it, they are putting a string in the sql statement and we are 'auto binding it'

You should

a) rewrite the sql in the application to use a bind
b) bind a RAW

then it will work.

short of that, when you compare datatype1 to datatype2 (you are comparing a RAW to a STRING), one of the types must be converted. Here the rule is "convert raw to string", you can see that easily;

ops$tkyte%ORA11GR1> create table t ( x raw(10) );

Table created.

ops$tkyte%ORA11GR1> @at
ops$tkyte%ORA11GR1> select * from t where x = '01020304';

Execution Plan
----------------------------------------------------------
Plan hash value: 1601196873

------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time
------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |     1 |     7 |     2   (0)| 00:00:01
|*  1 |  TABLE ACCESS FULL| T    |     1 |     7 |     2   (0)| 00:00:01
------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter(RAWTOHEX("X")='01020304')

Note
-----
   - dynamic sampling used for this statement


the rule is - we KNOW we can convert the raw into hex without error, we are not sure the string can be converted into RAW (that it is in hex) without error - so we do the natural thing - convert the raw to hex and compare.


Now, you have an index on that raw field, not on rawtohex() of that field.

So, your two options if you don't fix the really bad bug in the code are

a) index rawtohex(your_column) instead of indexing your column.
b) use hextoraw('your string') in your code

but if you do (b) you might as well FIX THE CODE


But please, do not say you are using binds, you are NOT. You have cursor sharing set to some value other than EXACT


ops$tkyte%ORA11GR1> alter session set cursor_sharing = force;

Session altered.
<b>
ops$tkyte%ORA11GR1> select * from t where x = '010101';
</b>
no rows selected

ops$tkyte%ORA11GR1> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------
SQL_ID  7287p5bwnfakf, child number 0
-------------------------------------
<b>select * from t where x = :"SYS_B_0"</b>

Plan hash value: 1601196873

------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time
------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |     2 (100)|
|*  1 |  TABLE ACCESS FULL| T    |     1 |     7 |     2   (0)| 00:00:01
------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
<b>
   1 - filter(RAWTOHEX("X")=:SYS_B_0)
</b>
Note
-----
   - dynamic sampling used for this statement


22 rows selected.

Rating

  (8 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

thatipa, October 30, 2008 - 7:11 pm UTC

Hi Tom,
      Thanks for your support. First thing is, the cursor_sharing is set to exact in the database. Also, I asked application people about bind variable. And they said,
In our application (JAVA), we are using PreparedStatement and passing binary value in a byte array using both setBytes() and setBinaryStream() to set the raw column value in the query. And, the raw column is the primary key. 

can you please explain me why is it not using index on streamchunkid column? If we use raw data type does it not use index on it? 

Please explain me clearly.

SQL> explain plan for SELECT chunkData FROM PackageStoreStreamChunk WHERE streamChunkId = :1;

Explained.

SQL> @$ORACLE_HOME/rdbms/admin/utlxpls.sql

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------

-----------------------------------------------------------------------------
| Id  | Operation         | Name                    | Rows  | Bytes | Cost  |
-----------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |                         |    91 | 14014 |    32 |
|   1 |  TABLE ACCESS FULL| PACKAGESTORESTREAMCHUNK |    91 | 14014 |    32 |
-----------------------------------------------------------------------------

SQL> show parameter cursor_sharing;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
cursor_sharing                       string      EXACT


Thanks,
thatip.

Tom Kyte
November 02, 2008 - 4:08 pm UTC

... First thing is, the cursor_sharing is set to
exact in the database. ...

not in the session that is showing this, how likely is it that your developers would use:

streamId= :"SYS_B_0"; 


??? come on...


... can you please explain me why is it not using index on streamchunkid column? ...

because you are NOT BINDING..

tell you what - you show me the snippet of code behind this... then, you'll see...

Your plan already shows you the answer

Dion Cho, October 30, 2008 - 10:15 pm UTC

See following

-- 10.2.0.1

select /*+ gather_plan_statistics */ *
from t1
where c1 = :b1
;

select * from table(dbms_xplan.display_cursor(null,null,'allstats last'));

--------------------------------------------------------------------------
| Id  | Operation         | Name | Starts | E-Rows | Cost (%CPU)| A-Rows |
--------------------------------------------------------------------------
|*  1 |  TABLE ACCESS FULL| T1   |      1 |      1 |    13   (8)|      0 |
--------------------------------------------------------------------------
                                                                          
Predicate Information (identified by operation id):                       
---------------------------------------------------                       
                                                                          
   1 - filter(RAWTOHEX("C1")=:B1)                                         


Oracle internally transforms c1 = :b1 to rawtohex(c1) = :b1.
For that reason, Oracle does not even consider the index.

You can do 10053 trace to verify what Oracle really does.

Full table scan issues..

Maverick, November 03, 2008 - 11:09 am UTC

Tom, Since this thread has started with a full table scan issue, I thought my question would fit here.

I am using Oracle 10g rel 2 and was trying to test the query performance.
Here is the table structure and other info.

All columns named as FK% are foreign keys to other tables

SQL> desc authentication   --> This is Parent Table 
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------------
 ID                                        NOT NULL NUMBER(10)        => Primary Key
 FK_CASE_ID                                         NUMBER(10)
 FK_REFERRAL_ID                                     NUMBER(10)
 FK_BUDGET_UNIT_ID                                  NUMBER(10)
 SERVICE_FISCAL_YEAR                                NUMBER(10)
 FK_CASELOAD_ID                                     NUMBER(10)        ==> Foreign key to another table
 AUTHENT_NUMBER                                     VARCHAR2(20)
 APROCE_TYPE                                        NUMBER(10)
 FK_PROVIDER_ID                                     NUMBER(10)
 ASA411_FUNDING_SOURCE                              VARCHAR2(8)
 FK_STATUS_STEP_ID                                  NUMBER(38)
 ASA411_PROVIDER                                    VARCHAR2(8)

SQL> select count(*) from authentication;

  COUNT(*)                                                                      
----------                                                                      
   1714923                                                                      



SQL> desc authent_status   -- Child table for AUTHENTICATION table
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------------
 ID                                        NOT NULL NUMBER(10)    ==> Primary Key
 FK_AUTHENT_ID                                      NUMBER(10)    => Foreign key to AUTHENTICATION Table
 ASSIGN_FROM_STATUS                                 NUMBER(10)
 ASSIGN_FROM_DATE                                   DATE
 ASSIGN_TO_STATUS                                   NUMBER(10)
 CURRENT_AUTHENT                                    VARCHAR2(1)
 STEPID                                             NUMBER(10)
 COMMENTS                                           VARCHAR2(200)
 MODIFIED_BY                                        NUMBER(10)
 MODIFIED_TIMESTAMP                                 TIMESTAMP(6)

SQL> select count(*) from Authent_status;

  COUNT(*)                                                                      
----------                                                                      
   3431488                                                                      

SQL> select count(*) from authent_status where assign_to_status=800 and current_authent='Y';

  COUNT(*)                                                                      
----------                                                                      
   1710618                                                                      

SQL> explain plan for
  2  select au.id,au.authent_number,st.assign_to_status
  3  from authentication au,
  4       authent_status st
  5  where au.id=st.fk_authent_id
  6   and st.assign_to_status=800
  7   and st.current_authent='Y';

Explained.

SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT                                                               
--------------------------------------------------------------------------------
Plan hash value: 3334870346                                                     
                                                                                
-------------------------------------------------------------------------------------------------                                                               
                                                                                
| Id  | Operation          | Name               | Rows  | Bytes |TempSpc| Cost (%CPU)| Time     |                                                               
                                                                                
-------------------------------------------------------------------------------------------------                                                               
                                                                                

PLAN_TABLE_OUTPUT                                                               
--------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |                    |   843K|    27M|       | 16848   (3)| 00:03:23 |                                                               
                                                                                
|*  1 |  HASH JOIN         |                    |   843K|    27M|    18M| 16848   (3)| 00:03:23 |                                                               
                                                                                
|*  2 |   TABLE ACCESS FULL| AUTHENT_STATUS     |   843K|  9057K|       |  5111   (5)| 00:01:02 |                                                               
                                                                                
|   3 |   TABLE ACCESS FULL| AUTHENTICATION     |  1714K|    37M|       |  7895   (3)| 00:01:35 |                                                               

PLAN_TABLE_OUTPUT                                                               
--------------------------------------------------------------------------------
                                                                                
-------------------------------------------------------------------------------------------------                                                               
                                                                                
                                                                                
Predicate Information (identified by operation id):                             
---------------------------------------------------                             
                                                                                
   1 - access("AU"."ID"="ST"."FK_AUTHENT_ID")                             
   2 - filter("ST"."ASSIGN_TO_STATUS"=800 AND "ST"."CURRENT_AUTHENT"='Y')          

16 rows selected.

Indexes :

Table Authent_Status:
---------------------
  Index Name           Unique? Logging Degree Column Name       Order Position Index Owner

 IDX_AUTH_STATUS_ID      N           YES 1 ASSIGN_TO_STATUS Asc 1            KTP
 IDX_STATUS_AUTHORIZATION     N           YES 1 FK_AUTHENT_ID    Asc 1            KTP
 PK_AU_STATUS       Y           YES 1 ID                 Asc 1     KTP


Table Authentication:
---------------------

  Index Name  Unique?  Logging Degree Column Name  Order Position Index Owner

 IDX_AUTH_CASELOAD    N  YES 1 FK_CASELOAD_ID  Asc 1  CMS
 IDX_AUTH_FK_CURRENT_CASE   N  YES 1 FK_CURRENT_CASE_ID Asc 1 CMS
 IDX_AUTH_REFERRAL    N  YES 1 FK_REFERRAL_ID  Asc 1 CMS
 IDX_TAU_AUTH_AUTH_NUM    Y  YES 1 AUTHENT_NUMBER  Asc 1 CMS
 PK_AUTHENTICATION    Y  YES 1 ID   Asc 1 CMS

Question is why it is doing full table scan on parent table. When it finds a record in child table and gets fk_authent_id, and this is indexed, shouldn't it use index?
I understand full table scan on child as it hits more than 5% of rows but why FTS on Parent table?
What else could be the reason for Full table scan? could it be the amount of data?
Anything you can predict with the given information wouldbe useful and can help me test different scenarios.

Thanks a Bunch.
Tom Kyte
November 10, 2008 - 3:15 pm UTC

http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:6749454952894#6760861174154


that plan is beautiful that you have for getting all of the rows as fast as possible.


you get 1.7 from one table. you do NOT want to do 1.7 million index range scans do you - think about the work involved here.

two juicy fast full scans
really performant hash join.


sounds great to me.

JAVA code for the query

thatipa, November 03, 2008 - 7:10 pm UTC

Hi Tom,
      i am providing the JAVA code behind this SQL query. 

GET_CHUNK_DATA_SQL  = ¿SELECT chunkData FROM PackageStoreStreamChunk WHERE streamChunkId = ?¿;
            conn = getConnection();
            ps = conn.prepareStatement(GET_CHUNK_DATA_SQL);
            ps.setBytes(1, getPk().getId());
            rs = ps.executeQuery();
            if (!rs.next()) {
            throw new IllegalStateException(¿)
            }
            data = rs.getBytes(1);

One more question:

If I use Index hint in the SQL query then, does it help in doing index scans? The index is built on another column (originaloffset) 

Ex:-
This is the index hint version. 

SQL> SELECT /*+ index(PackageStoreStreamChunk IDX_PACKAGESTORESTREAMCHUNKSTREAMID_ORIGINALOFFSET)*/ chunkData FROM PackageStoreStreamChunk WHERE streamChunkId = ?¿;

Thanks,
~Thatipa.

Tom Kyte
November 11, 2008 - 11:56 am UTC

read the link I provided above.

full table scans..

Maverick, November 11, 2008 - 9:43 am UTC

Tom, that was a good point.
But, Doesn't the execution go like this:

Fetch a row from "authent_status" table and take fk_authent_id from it and go to "authentication" table and look for the corresponding record. So if that's the case ,then where is question of 1.7 mill at that point. Shouldn't it be better of using index and fetch parent without doing full table scan?
It's correct in fetching 3.4 mill without indexes but each row from this table should use index to fetch from parent table. Shouldn't it work this way?


Tom Kyte
November 11, 2008 - 4:34 pm UTC

think about this.....


the one full scan returns 1.7 million rows (authent_status)

the other full scan returns 3.4 million rows

SQL> select count(*) from Authent_status;

COUNT(*)
----------
3431488

If we use 1.7 million rows to do an index range scan 1.7 million times that is bad.

If we use 3.4 million rows to do an index range scan 3.4 million times that is worse


check out the link - if you want to get A LOT of data joined to A LOT of data - quickly - indexes are "not good" in general.




please - do this:


loop
   say to yourself indexes are not all goodness
   say to yourself full scans are not all evil
   exit when (you believe those two statements are true)
end loop;







Bind variable usage after index creation

thatipa, November 19, 2008 - 3:24 pm UTC

Hi Tom,
       I posted the JAVA code as you asked. I am waiting for response. 
Also, I created index with rawtohex() on the column streamchunkid.

I created an index and tested the performance improvement in downloading. I saw improvement but the strange thing here is, I tried  to compare the improvement in performance with and with out index. So, I dropped the index and tested the download but surprisingly the performance was same. Why is it happening like this? 

I saw a change in the bind variable used by the oracle. Before and after creating index. The first query is the one I posted for the first time here and I saw change in the bind variable usage later after creating index. 
What do you say about this?

Before:
SELECT  chunkData FROM PackageStoreStreamChunk where streamChunkId= :"SYS_B_0";

After:
SELECT chunkData FROM PackageStoreStreamChunk WHERE streamChunkId = :1

Even after dropping the index, I saw the same query as above  in the AWR report.

SQL> SELECT chunkData FROM PackageStoreStreamChunk WHERE streamChunkId = :1

Thanks,
thatip.

Tom Kyte
November 24, 2008 - 11:04 am UTC

sorry, but I don't believe you

you say "cursor sharing is not set to force/similar"

but it is "SELECT streamId, originalOffset, originalSize, chunkData FROM PackageStore where streamId= :"SYS_B_0"; "

CLEARLY it is. That is obvious - that would be pretty much the reason sys_b_0 is there.

so, I question your sample. I don't believe it - OR - I don't believe the original question.

so, get us a TKPROF from that example - it will look very different from what you posted originally.

Same issue with RAW PK and Full Table Scans here

Desmond, February 13, 2009 - 4:56 pm UTC

We are on 10.2.0.4. CURSOR_SHARING is set to EXACT. Our table has a primary key that is defined as RAW(16). From the SQL Tuning Advisor, we see the following plan:

SQL ID : gzn5g71qtknt3
SQL Text : SELECT ATTRID, SVCID, SEQNUM, ATTRTYPE, ATTRNAME FROM GENERALSERVICEATTRIBUTE WHERE ATTRID = :1

Plan hash value: 2145337433

---------------------------------------------------------------------------------------------

| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |

---------------------------------------------------------------------------------------------

| 0 | SELECT STATEMENT | | 1 | 54 | 11738 (4)| 00:02:21 |

|* 1 | TABLE ACCESS FULL| GENERALSERVICEATTRIBUTE | 1 | 54 | 11738 (4)| 00:02:21 |

---------------------------------------------------------------------------------------------



Predicate Information (identified by operation id):
---------------------------------------------------

1 - filter(RAWTOHEX("ATTRID")=:1)


The tuning advisor then advises use to create a RAWTOHEX index of the ATTRID column, which is the primary key.

What else could cause Oracle to do a full table scan?
Tom Kyte
February 16, 2009 - 12:07 pm UTC

your programmers - they bound a varchar (string), not raw data.


so, go back to the code and tell your developers to either

a) hexttoraw their bind, where attrid = hextoraw(?)
b) bind the right datatype in the first place


ops$tkyte%ORA10GR2> create table t ( x raw(16) primary key, y char(2000) );

Table created.

ops$tkyte%ORA10GR2>
ops$tkyte%ORA10GR2> exec dbms_stats.set_table_stats( user, 'T', numrows => 1000000 );

PL/SQL procedure successfully completed.

ops$tkyte%ORA10GR2>
ops$tkyte%ORA10GR2> declare
  2          l_varchar varchar2(32);
  3          l_raw     raw(16);
  4  begin
  5          for x in (select * from t query_VARCHAR where x = l_varchar )
  6          loop
  7                  null;
  8          end loop;
  9          for x in (select * from table(dbms_xplan.display_cursor) )
 10          loop
 11                  dbms_output.put_line( x.PLAN_TABLE_OUTPUT );
 12          end loop;
 13
 14
 15          for x in (select * from t query_RAW where x = l_raw )
 16          loop
 17                  null;
 18          end loop;
 19          for x in (select * from table(dbms_xplan.display_cursor) )
 20          loop
 21                  dbms_output.put_line( x.PLAN_TABLE_OUTPUT );
 22          end loop;
 23  end;
 24  /
SQL_ID  8w5zv3a6qj9s4, child number 0
-------------------------------------
SELECT * FROM T QUERY_VARCHAR WHERE X = :B1
Plan hash value: 1601196873
--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |   181 (100)|          |
|*  1 |  TABLE ACCESS FULL| T    | 10000 |   976K|   181  (83)| 00:00:01 |
--------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter(RAWTOHEX("X")=:B1)
SQL_ID  85zj81s4gcac3, child number 0
-------------------------------------
SELECT * FROM T QUERY_RAW WHERE X = :B1
Plan hash value: 1318105761
--------------------------------------------------------------------------------------------
| Id  | Operation                   | Name         | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |              |       |       |     2 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T            |     1 |   100 |     2   (0)| 00:00:01 |
|*  2 |   INDEX UNIQUE SCAN         | SYS_C0034611 |     1 |       |     1   (0)| 00:00:01 |
--------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - access("X"=:B1)

PL/SQL procedure successfully completed.

Solution and explanation from ibm

JohnCrayton, March 18, 2011 - 5:29 pm UTC

Tom Kyte
March 18, 2011 - 6:43 pm UTC

explain how that is any different from anything said here.

basically it boils down to:

bind the right type.


No one here suggested creating a function based index on hextoraw. No where, No one - so what was the point?

More to Explore

CLOB/BLOB/etc

Complete documentation on Securefiles and Large Objects here