Skip to Main Content

Breadcrumb

Question and Answer

Tom Kyte

Thanks for the question, Connie.

Asked: January 12, 2011 - 3:28 pm UTC

Last updated: November 01, 2013 - 7:58 pm UTC

Version: 10.2.0.4

Viewed 10K+ times! This question is

You Asked

Dear Tom

Is 11.2.0.2 compression better than 10.2.0.4 ? if so how ?

Thanks
C

and Tom said...

In Oracle 9iR2 and later, with enterprise edition, you have what is known as 'basic compression' as a feature in the database. Basic compression works at the block level, removing redundant bits of information on a database block into the block header and storing them once per block - not once per occurrence on the block. This allows us to store 2x, 3x, 4x or more data per block. It only works during direct path operations such as insert /*+APPEND*/, alter table t move, create table as select, sqlldr direct=y. It does not PREVENT you from using normal insert/update/delete statements - it just means that the results of those statements will result in some non-compressed data. A single table may have some blocks compressed and some blocks not compressed - that works fine. There are restrictions as to what you can do with a basic compressed table as far as dropping columns and the like.

In Oracle 11g Release 1 and above there is the new advanced compression option. This option allows for normal, conventional path operations to be compressed - so a transactional application may save it's data in a compressed format without using direct path operations. It also removes most of the restrictions associated with basic compression such as a 255 column limit on compressed tables, scheema modifications, and the like.

So, the original basic compression still exists in 11g for enterprise edition users, and there is the new advanced compression that works in many more cases.

Is it "better"? It is the same sort of compression - it just works in more situations with fewer restrictions - so in that sense, yes it is "better"

Rating

  (15 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

A reader, January 14, 2011 - 9:09 am UTC

Tom,
This is excellent summary of what basic and advanced compression does. Can you please briefly explain what is hybrid columnar compression?

Thanks...
Tom Kyte
January 14, 2011 - 12:17 pm UTC

Hybrid columnar compression is performed by Exadata only - so it does not apply to "regular Oracle" - just the Exadata Database Machine.

Basically - we store data in what is called a compression unit - a set of blocks. So, unlike basic and advanced compression - which both work block by block and each block is 'self contained', you need only the one block to decompress your data - hybrid columnar compression works on a set of blocks - you need more than one block to decompress the data (we use 4x8k blocks typically).

The hybrid columnar compression (HCC) works with direct path loads only - similar to basic compression. The big difference is how hybrid columnar compression stores the data. Basic and advanced compression store data row by row - so row 1 comes on the block before row 2 and so on. HCC takes the data and before putting it on the block, factors out the data into arrays of columns. So, we'll take a stream of data and store all of the values for column 1 in array 1 and column 2 in array 2 and so on.

Before writing these arrays of column values out - one after the other - we go through a de-duplication process (the odds of repeating data are greatly increased since we are looking at at least 4 times as much data - spread over 4 blocks not just one). We can also do leading edge compression of the data in these arrays. We can also do more compression after the leading edge (we have two levels of compression available - you choose how much time you want to spend compressing data). Once we've compressed the columns as much as possible - we write them out to the compression unit. We write them out column by column now - not row by row.

In order to decompress - you need the compression unit - not just a single block.

A reader, January 18, 2011 - 11:22 am UTC

Thanks Tom. This is the best explanation of HCC that I have read. Our Exadata server is on order and I was trying to read up on its features. I was unable to grasp the "columnar" part of the HCC but your explanation has made it very clear.

Moving to Advanced compression from Basic compression

Dan, January 19, 2011 - 2:14 pm UTC

Hi Tom:

We have a 10.2.0.4 Reporting database where all tables are compressed partitioned tables. All the table/index partitions are compressed using basic compression option.
We are planning to upgrade to 11gR2.

I have Question with respect to Advanced compression:

a) If we want to use Advanced Compression, What steps are required to move from Basic Compression to Advanced Compression.

Do we need to uncompress all compressed tables and restart from the scratch to use advanced compression

Can you please list out the exact steps if you can. This would be very helpful. We know that Advanced compressions are licensed.
Tom Kyte
January 24, 2011 - 5:45 am UTC

index partitions are compressed using index key compression - predates the basic segment compression - it came out in 8.1.5, segment compression (which comes 'basic' or 'advanced') is 9ir2 and 11gr1.


a) none, basic compression is a 100% subset of advanced. If you want to ENABLE advanced (OLTP) compression on a table, you would simply alter it - all SUBSEQUENT DML will be compressed in addition to direct path operations.

If you have tables that are currently full of uncompressed data - you would need to reorganize them to make them have only compressed data.


what about HCC

Amir Riaz, February 01, 2011 - 6:36 am UTC

hi tom,

if in Exadata i have a table which is in HCC and OLTP compression with active partition in OLTP. at the end of month, can i alter OLTP compression to HCC and add new OLTP compression for next month. does HCC compression rate will suffer.
Tom Kyte
February 01, 2011 - 5:11 pm UTC

you would not alter it so much as "reorganize it", the partition would be rewritten using an alter move command. It would not affect the compression rate since the data would in effect be reloaded.

okay the steps to perform

Amir Riaz, February 02, 2011 - 2:40 am UTC

Hi tom,

so the steps to perform is

alter table partition from oltp compression to HCC compression

alter table move.

I am asking you this question because, with HCC you have to insert data in bulk using direct loading i.e

insert /*+ append*/ into target select * from source.

an insert into target values ( x,s,s); will insert data into a segament which will be a oracle block or OLTP compress block.

is, alter table target move will reorganise the data in HCC form.
Tom Kyte
February 02, 2011 - 7:44 am UTC

you have to use DIRECT PATH operations such as:

sqlldr direct=y
insert /*+append*/
create table as select
alter table move

any direct path operation would work fine.

Direct path loads for HCC

A reader, March 11, 2011 - 11:18 am UTC

Tom,
For Exadata HCC, is data pump import over network link considered direct path data load?

Thanks...

Tom Kyte
March 11, 2011 - 3:36 pm UTC

as long as it is using direct path - it can use hybrid columnar compression - HCC (it is best to always spell out an acronym the first time, even if you assume the person reading the text 'knows' what it means...)


http://docs.oracle.com/docs/cd/E11882_01/server.112/e16536/dp_overview.htm#SUTIL804

A reader, June 22, 2011 - 9:10 am UTC

Hi Tom

Thank you very much for the clear explanation of the Hybrid columnar compression.

you mentioned that the HCC will only be used when direct load is done. Let us say after direct load, If I perform updates on some rows. Whether the HCC will still be used or will those rows will not be using the HCC. If those rows will not be using the HCC then, How can we put them back to HCC.

Thanks in Advance

Raghavendra

Tom Kyte
June 22, 2011 - 10:48 am UTC

the modified rows will not be columnar compressed after the update.

to get them back, you need to direct path the data again, using alter table t move for example - to rebuild it.

Advanced compression on data and index

Sita, July 26, 2011 - 8:48 pm UTC

Hi Tom:

Oracle Ver 11.2.0.1

All documentation talk about Advanced compression on table data.
All our tables are OLTP compressed using "compression for all operations". This is still under POC phase. After OAC, we have saved space usage by 5x. Now the question is, Is it recommended to compress indexes also ?

Or index rebuild without compression is sufficient.

Thanks.
Tom Kyte
July 28, 2011 - 6:56 pm UTC

You can certainly look at using index key compression for your non-unique single column indexes OR any multi-column (concatenated) indexes.

You would probably want to use

alter index index_name validate structure;

in a test environment (with a full copy of your data) to identify indexes that would be compressable. For example:

ops$tkyte%ORA11GR2> create table t as select * from all_objects;

Table created.

ops$tkyte%ORA11GR2> 
ops$tkyte%ORA11GR2> create index t_idx1 on t(object_id);

Index created.

ops$tkyte%ORA11GR2> create index t_idx2 on t(object_type,owner,object_name);

Index created.

ops$tkyte%ORA11GR2> 
ops$tkyte%ORA11GR2> analyze index t_idx1 validate structure;

Index analyzed.

ops$tkyte%ORA11GR2> select name,  blocks, OPT_CMPR_COUNT, OPT_CMPR_PCTSAVE from index_stats;

NAME                               BLOCKS OPT_CMPR_COUNT OPT_CMPR_PCTSAVE
------------------------------ ---------- -------------- ----------------
T_IDX1                                256              0                0

<b>that shows that t_idx1 is NOT compressable - it wouldn't be useful to use compress on it, however</b>

ops$tkyte%ORA11GR2> analyze index t_idx2 validate structure;

Index analyzed.

ops$tkyte%ORA11GR2> select name,  blocks, OPT_CMPR_COUNT, OPT_CMPR_PCTSAVE from index_stats;

NAME                               BLOCKS OPT_CMPR_COUNT OPT_CMPR_PCTSAVE
------------------------------ ---------- -------------- ----------------
T_IDX2                                640              2               28

<b>that index would benefit from compress 2 - it would be at least 28% smaller</b>

ops$tkyte%ORA11GR2> alter index t_idx2 rebuild compress 2;

Index altered.

ops$tkyte%ORA11GR2> analyze index t_idx2 validate structure;

Index analyzed.

ops$tkyte%ORA11GR2> select name,  blocks, OPT_CMPR_COUNT, OPT_CMPR_PCTSAVE from index_stats;

NAME                               BLOCKS OPT_CMPR_COUNT OPT_CMPR_PCTSAVE
------------------------------ ---------- -------------- ----------------
T_IDX2                                384              2                0

<b>and it is...</b>



bear in mind that validate structure LOCKS the index - do NOT do this in your live OLTP system!

Exadata Hybrid Columnar Compression

A reader, January 09, 2013 - 11:22 am UTC

Tom,
In Exadata, does HCC happen on the database server or in storage server? How does smart scan work with HCC? Are the blocks uncompressed in storage server, smart scanned and only few matching rows returned to database server?

Thanks...

Tom Kyte
January 14, 2013 - 12:59 pm UTC

trick question - both.


the database server can process cached HCC blocks. So if we processed the data and compressed it 50:1, you'd be able to cache 50 times as much of that data in the cache since the compressed block would be cached in the SGA.

the storage array can process those blocks as well - it can cache them in the flash cache - it can read and process them with query offloading.


If we smart scan it - then the data is read compressed by the storage server, it is decompressed on the storage server, the rows and columns of interest are discovered and sent to the database layer.

If we read it "conventionally" (no smart scan) then the storage array just transmits the entire compressed block to the database layer.

255 columns restriction still present with advanced compression

Josef, September 18, 2013 - 11:15 am UTC

You said:
--
It (advanced compression) also removes most of the restrictions associated with basic compression such as a 255 column limit on compressed tables.
--

Based on the 11.2 advanced compression documentation;
as well as, based on a simple testcase, with a 1000 columns table, that is not true.
- dba_tables.compression/compression_for says (ENABLED,OLTP), but the table (after "move compress for oltp") still takes up the same amount of blocks, as before compression (while a testcase, of the same dummy data, in a 255-column table, shows a block number reduction factor of 15, when going from NOCOMPRESS to COMPRESS FOR OLTP).
Tom Kyte
September 24, 2013 - 11:22 pm UTC

sorry, you are correct (i have to verify on 11.2.0.4 however). It is in 12.1 though for sure:

ops$tkyte%ORA12CR1> declare
  2          l_string clob := 'create table t ( x int';
  3  begin
  4          for i in 1 .. 999
  5          loop
  6                  l_string := l_string || ', c' || to_char(i) || ' varchar2(20) default rpad( ''x'', 20, ''x'' )';
  7          end loop;
  8          l_string := l_string || ' )';
  9          execute immediate l_string;
 10  end;
 11  /

PL/SQL procedure successfully completed.

ops$tkyte%ORA12CR1>
ops$tkyte%ORA12CR1> insert into t (x) select rownum from all_users;

37 rows created.

ops$tkyte%ORA12CR1>
ops$tkyte%ORA12CR1> exec show_space( 'T' );
Unformatted Blocks .....................              62
FS1 Blocks (0-25)  .....................               0
FS2 Blocks (25-50) .....................               1
FS3 Blocks (50-75) .....................               0
FS4 Blocks (75-100).....................              34
Full Blocks        .....................             147<b>
Total Blocks............................             256
Total Bytes.............................       2,097,152
Total MBytes............................               2</b>
Unused Blocks...........................               0
Unused Bytes............................               0
Last Used Ext FileId....................               4
Last Used Ext BlockId...................           3,456
Last Used Block.........................             128

PL/SQL procedure successfully completed.

ops$tkyte%ORA12CR1> alter table t compress for oltp;

Table altered.

ops$tkyte%ORA12CR1> alter table t move;

Table altered.

ops$tkyte%ORA12CR1> exec show_space( 'T' );
Unformatted Blocks .....................               0
FS1 Blocks (0-25)  .....................               0
FS2 Blocks (25-50) .....................               0
FS3 Blocks (50-75) .....................               0
FS4 Blocks (75-100).....................               0
Full Blocks        .....................             108
Total Blocks............................             120<b>
Total Bytes.............................         983,040
Total MBytes............................               0</b>
Unused Blocks...........................               2
Unused Bytes............................          16,384
Last Used Ext FileId....................               4
Last Used Ext BlockId...................           3,808
Last Used Block.........................               6

PL/SQL procedure successfully completed.

A reader, September 24, 2013 - 8:24 pm UTC

We have a 10g database that is currently about 3TB in size and we are planning to upgrade this database to 11g. We would like to compress the data in the 10g database and are trying to decide on the following

1. What method do we use to compress the 10g data ?
2. Do we compress in 10g and then move to 11g or move to 11g and then compress in 11g.

Please help.

HCC and database blocks

Paul, September 30, 2013 - 5:11 pm UTC

In hybrid columnar compression segments, data is stored in a different kind of "blocks" than I am familiar with. How should I interpret block related concepts like PCTFREE, PCTUSED, the "blocks" column in user_segments and user_extents, and chained rows in HCC segments?

Advanced compression datatypes

Jeff, October 16, 2013 - 9:34 pm UTC

Tom,

Does 11.2 advanced compression work with columns of CHAR datatype? I haven't been able to find any documentation saying it does not, but in my first experiment with trying to use AC on a legacy application table, I'm not getting any reduction in size. And I know there are a lot of embedded spaces in the data.

I did an experiment changing the datatypes to VARCHAR2 and reduced the avg_row_len from 2137 to 128.

Unfortunately there is a FUD factor preventing reworking the app with the new datatype, but since AC is supposed to be transparent to the application I thought it would be a good compromise.

Tom Kyte
November 01, 2013 - 7:58 pm UTC

well, compression is going to be harder with char(2000).

right now, you are getting a max if 3 rows per block. that is the most that can fit (assume 8k block).

Now you employ compression.

IF you insert two rows into a block with the same char(2000) value, you'll be able to get two more rows at least on that block (so four). If the third row on that block is the same - you'll be able to get probably five rows.

but - you'd have to be inserting the same value over and over for this to work. and then move onto the next value.

that is - if three people inserted one after the other and they inserted three different values - the block they fill would be full - we'd move onto the next block. There would be NO compression effectively.


however, if all three inserted the same value - that would compress nicely. But this isn't real world. there might be duplication in that field - but it is unlikely it would realistically compress.



Lets get to the FUD - let me help you there. I have a rule "never ever ever ever - as in never - use CHAR, never, not even for CHAR(1)". I'll defeat any FUD they can conjure up.


CHAR compression

Jeff, May 08, 2014 - 8:42 pm UTC

But shouldn't all those padded spaces on the end of each column compress down and allow more rows to fit in the block?

Licensing for Oracle Compression: Oracle 10g Enterprise and Oracle 11g Enterprise

Vijay, June 04, 2014 - 4:47 pm UTC

Hello Tom,

With regards to Compression options available on Oracle 10g and 11g Enterprise Editions, need a clarity as to which feature require additional licensing and which doesn't for both 10g and 11g versions.

For instance, in 11g Enterprise, does the below listed Compression features involve additional licensing costs?

- Basic Compression
- OLTP Compression
- SecureFiles Compression
- Index Compression
- Backup Compression
- Oracle Exadata hybrid columnar compression/data warehousing
- Oracle Exadata hybrid columnar compression/ archiving

I'm looking for a comprehensive list of Compression options available on both 10g and 11g Enterprise Editions and the Licensing Costs associated with each of those. Appreciate if you can help me with this information?

Thanks in advance