Skip to Main Content

Breadcrumb

Question and Answer

Connor McDonald

Thanks for the question, Swapnil.

Asked: May 22, 2017 - 6:29 pm UTC

Last updated: May 26, 2017 - 1:11 am UTC

Version: 11g

Viewed 1000+ times

You Asked

Hi TOM Team,

I am having a Huge table containing more the 100 Billion Records . The table is containing the transactions of more than 6 years and the table Structure is as follows :

Table Name : FCT_TRANS

Table Structure :

ACID varchar(20), TRAN_DATE date, TRAN_TYPE varchar(10), TRAN_SUB_TYPE varchar2(10), PART_TRAN_TYPE varchar2(10), BRCH_ID varchar2(10), TRAN_RMKS varchar2(100), TRANID number, TRAN_AMT number etc.

Indexed Columns : Composite Index ( ACID, TRAN_DATE), Normal Index - TRAN_TYPE.

There is one more table having around 25 Millions Record and is the master table of accouts live in the system.

Table Name : DIM_ACCT

Table Structure :
ACID varchar(20), ACT_OPN_DATE date, SCHM_TYPE varchar(10), SCHM_CODE varchar2(10), ACT_NAME varchar2(50), BRCH_ID varchar2(10), etc.

Indexe Columns : ACID

Now both the tables are system tables and ACID is the mapping key between the two tables.

I want to extract Monthly Summary for Transaction Type-Wise, Sub-type wise , SCHM_TYPE wise Transaction Count and Transaction Amout for the Last Two Years.

Sample OP Format :

MONTH , TRAN_TYPE, TRAN_SUB_TYPE, PART_TRAN_TYPE, SCHM_TYPE, SUM_OF_TRAN_AMT, Count_of_Transactions, Count_of_Uniq_Transacting_Acts

We will have to join the two tables on the basis of ACID for the requirement an will have to take monthly summary for the last two years. Query executed for the requirement is provided below :

Current Query Execution :

select to_char(a.tran_date,'MONYY') TRAN_DT,
a.TRAN_TYPE,a.TRAN_SUB_TYPE, a.PART_TRAN_TYPE, b.SCHM_TYPE,
sum(a.TRAN_AMT) SUM_OF_TRAN_AMT,
count(a.acid) Count_of_Transactions,
count(distinct a.acid) Count_of_Uniq_Transacting_Acts
from FCT_TRANS A, DIM_ACCT B
where a.acid=b.acid
and a.tran_date between '01-11-2016' and '30-11-2016'
group by to_char(a.tran_date,'MONYY'),a.TRAN_TYPE,a.TRAN_SUB_TYPE, a.PART_TRAN_TYPE, b.SCHM_TYPE;

We have tried to extract these data by using simple join and group by but it is taking more than 6-7 Hours for on month only. So, the complete data extraction would take around 6-7 days.

Please help with an optimized process for extracting such data requirement at the earliest .

Also, please provide some sample scripts and optimization tricks to tackle such requirements as I am new to the Oracle Technology and want to learn.


and Connor said...

Options:

1) partition the table on tran_date

Obviously takes a large degree of testing because of potential impact on your existing application code

2) pre-prepared summaries

Create materialized views of that summary data so your queries can access a smaller set of data.

3) More horsepower

Look at using parallel query to get the answers more quickly.

4) Less data

Look at opportunities to compress some or all of the data to make the query source data sets smaller.

Rating

  (2 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

Swapnil Bendale, May 25, 2017 - 5:49 pm UTC

Hi Sir,

Thank you for your kind advices. But most of them either is now possible at my end due to having limited access or are already present .

Request you to provide some optimised PL SQL Examples to tackle such scenarios. As I am new to the technology , It would be great help if you can provide some practical optimised examples.


Swapnil Bendale, May 25, 2017 - 5:49 pm UTC

Hi Sir,

Thank you for your kind advices. But most of them either is now possible at my end due to having limited access or are already present .

Request you to provide some optimised PL SQL Examples to tackle such scenarios. As I am new to the technology , It would be great help if you can provide some practical optimised examples.


Connor McDonald
May 26, 2017 - 1:11 am UTC

Show us your testing where you

- considered each of the options suggested
- where it worked or didnt worked
- what obstacles you hit

because it doesnt really cut it when you say "we want it faster, but we dont want to try things that might make it faster"