Skip to Main Content
  • Questions
  • What is efficient way to paginate on the large table?

Breadcrumb

Question and Answer

Connor McDonald

Thanks for the question, Ashish.

Asked: June 30, 2019 - 3:20 pm UTC

Last updated: July 02, 2019 - 6:20 am UTC

Version: 12

Viewed 10K+ times! This question is

You Asked

Hi Tom,

We are using the following query in java application, for getting the query results in the pagination. To speed up the task we are executing 15 queries at the same time. The queries are being executed on different machines but targeting the same Table in the database. Please Note: In our case, we are only paginating those Tables having the primary key and we are not using any joins between Tables.

SELECT 
  subquery.id, subquery.column1 ... 
  FROM
    (
      SELECT 
        large_rows.id, large_rows.column1, ... ROW_NUMBER() 
          OVER 
            (ORDER BY id ASC) 
              AS 
                row_number 
                  FROM 
                    <table_name>
                      ) 
                        subquery 
                          WHERE 
                            row_number 
                              BETWEEN 
                                <offset+1> 
                                  AND 
                                    <lenght>;



when these queries are being executed against large Table having the more than 2 billion records in that then we are getting the following error:

ORA-01652: unable to extend temp segment in tablespace  


And when we are trying to execute the query on SQL DEVELOPER, it seems to be taken a long time, even for the starting pages so we changed the above query by the following query:

SELECT /*+ ordered use_nl(p s) */
    s.id, s.column1...
FROM
    (
        SELECT
           ROWNUM RN,
           RD
        FROM
            (
                SELECT
                    ROWID RD
                FROM
                    <table_name>
               ORDER BY
               <table_name>.id
            ) T
        WHERE
            ROWNUM < 2000
    ) P,
    large_rows S
WHERE
    RN > 1
    AND P.RD = S.ROWID; 


This query executes faster then the previous one.

Can you please suggest will this query work against ORA-01652? Is any better query then the above in this situation? Can we paginate query results without using order by clause (thinking because of the order by clause may cause this issue again)?

and Connor said...

We need to know what execution plans are being used.

This video will show you what to look for



Rating

  (3 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

on the Index in ID Column

Rajeshwaran Jeyabal, July 02, 2019 - 7:33 am UTC

If "ID" is a key column, then we dont need to sort the entire results sets in the table.
the index is nicely sorted already and the optimizer will make use of those index on the ID column to answer these paginated queries.

demo@PDB1> set autotrace traceonly explain
demo@PDB1> select *
  2  from (
  3  select b.* ,
  4      row_number() over(order by id) r
  5  from big_table b
  6       )
  7  where r between 51 and 100;

Execution Plan
----------------------------------------------------------
Plan hash value: 63938039

----------------------------------------------------------------------------------------------
| Id  | Operation                     | Name         | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT              |              |   100 | 50700 | 23801   (1)| 00:00:01 |
|*  1 |  VIEW                         |              |   100 | 50700 | 23801   (1)| 00:00:01 |
|*  2 |   WINDOW NOSORT STOPKEY       |              |  1000K|   118M| 23801   (1)| 00:00:01 |
|   3 |    TABLE ACCESS BY INDEX ROWID| BIG_TABLE    |  1000K|   118M| 23801   (1)| 00:00:01 |
|   4 |     INDEX FULL SCAN           | BIG_TABLE_PK |  1000K|       |  2098   (1)| 00:00:01 |
----------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("R">=51 AND "R"<=100)
   2 - filter(ROW_NUMBER() OVER ( ORDER BY "ID")<=100)

demo@PDB1> select *
  2  from big_table
  3  order by id
  4  offset 50 rows fetch next 50 rows only;

Execution Plan
----------------------------------------------------------
Plan hash value: 63938039

----------------------------------------------------------------------------------------------
| Id  | Operation                     | Name         | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT              |              |  1000K|   495M| 23801   (1)| 00:00:01 |
|*  1 |  VIEW                         |              |  1000K|   495M| 23801   (1)| 00:00:01 |
|*  2 |   WINDOW NOSORT STOPKEY       |              |  1000K|   118M| 23801   (1)| 00:00:01 |
|   3 |    TABLE ACCESS BY INDEX ROWID| BIG_TABLE    |  1000K|   118M| 23801   (1)| 00:00:01 |
|   4 |     INDEX FULL SCAN           | BIG_TABLE_PK |  1000K|       |  2098   (1)| 00:00:01 |
----------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("from$_subquery$_002"."rowlimit_$$_rownumber"<=CASE  WHEN (50>=0) THEN
              50 ELSE 0 END +50 AND "from$_subquery$_002"."rowlimit_$$_rownumber">50)
   2 - filter(ROW_NUMBER() OVER ( ORDER BY "BIG_TABLE"."ID")<=CASE  WHEN (50>=0) THEN
              50 ELSE 0 END +50)

@Rajeshwaran Jeyabal

Stew Ashton, July 02, 2019 - 9:04 am UTC

With your proposal, Oracle will still visit the table for the rows before the offset.
create table t (
  pk number primary key,
  x varchar2(4000)
) nologging;
insert /*+ append*/ into t
select rownum, rpad(rownum,100,'-') from dual
connect by rownum <= 500000;
commit;

select /*+ gather_plan_statistics */ * from t
order by pk
offset 1000 rows fetch next 10 rows only;
 
------------------------------------------------------------------------------------
| Id  | Operation                     | Name  | Starts | E-Rows | A-Rows | Buffers |
------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT              |       |      1 |        |     10 |      20 |
|*  1 |  VIEW                         |       |      1 |   1010 |     10 |      20 |
|*  2 |   WINDOW NOSORT STOPKEY       |       |      1 |   1010 |   1010 |      20 |
|   3 |    TABLE ACCESS BY INDEX ROWID| T     |      1 |    500K|   1010 |      20 |
|   4 |     INDEX FULL SCAN           | SYS_* |      1 |   1010 |   1010 |       4 |
------------------------------------------------------------------------------------
Notice in step 3 the table is visited 1010 times. Also, when the offset is big then Oracle does a full table scan.
select /*+ gather_plan_statistics */ * from t 
order by pk 
offset 100000 rows fetch next 10 rows only;
 
---------------------------------------------------------------------------------------
| Id  | Operation                | Name | Starts | E-Rows | A-Rows | Buffers | Reads  |
---------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT         |      |      1 |        |     10 |    7702 |   1246 |
|*  1 |  VIEW                    |      |      1 |    100K|     10 |    7702 |   1246 |
|*  2 |   WINDOW SORT PUSHED RANK|      |      1 |    500K|    100K|    7702 |   1246 |
|   3 |    TABLE ACCESS FULL     | T    |      1 |    500K|    500K|    7702 |   1246 |
---------------------------------------------------------------------------------------
A workaround is to get the ROWIDs in a subquery:
select /*+ gather_plan_statistics */ * from t
where rowid in (
  select rowid rid from t
  order by pk
  offset 100000 rows fetch next 10 rows only
);

-------------------------------------------------------------------------------------
| Id  | Operation                   | Name     | Starts | E-Rows | A-Rows | Buffers |
-------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |          |      1 |        |     10 |     190 |
|   1 |  NESTED LOOPS               |          |      1 |      1 |     10 |     190 |
|   2 |   VIEW                      | VW_NSO_1 |      1 |    100K|     10 |     189 |
|   3 |    HASH UNIQUE              |          |      1 |      1 |     10 |     189 |
|*  4 |     VIEW                    |          |      1 |    100K|     10 |     189 |
|*  5 |      WINDOW NOSORT STOPKEY  |          |      1 |    100K|    100K|     189 |
|   6 |       INDEX FULL SCAN       | SYS_C00* |      1 |    500K|    100K|     189 |
|   7 |   TABLE ACCESS BY USER ROWID| T        |     10 |      1 |     10 |       1 |
-------------------------------------------------------------------------------------
Please note that I ran this in version 19.3. If memory serves, previous versions did not always get the same plan. Not only that, but the OFFSET and FETCH clauses were not optimized in previous versions, so we had to fall back on the inline views with ROWNUM!

Best regards,
Stew Ashton

To Stew : on deep down the pages

Rajeshwaran Jeyabal, July 02, 2019 - 2:45 pm UTC

Nice input.

But when doing pagination - we hardly visit no more than 5 pages(provided 25 or 50 rows per page).
if we dont get the required results with in the navigation of few pages, in real-world we dont page-down, instead we would redefine the search criteria.

so offset 100K rows and fetch 10 rows is not realistic.

More to Explore

Analytics

Analytic SQL got you confused? Check out Connor McDonald's complete video course.