Skip to Main Content
  • Questions
  • Transparent Data encryption in 10g release 2

Breadcrumb

Question and Answer

Tom Kyte

Thanks for the question, Raj.

Asked: July 21, 2005 - 11:48 pm UTC

Last updated: August 02, 2012 - 9:00 am UTC

Version: 10.2.x

Viewed 10K+ times! This question is

You Asked

Tom,
In your presentation at OGUC, you highlighted Transparent Data Security as one of the cool features available in 10g R2. Could you please shed some light on the following questions ?

* Who makes the encryption transparent in oracle (I mean which process)?
* How are the encryption keys (also the paraphrase to the keys) handled/maintained by Oracle?
* In which edition of Oracle this feature is available ?

BTW, it was an excellent and enjoyable presentation. I think your session had the maximum turn out compared to rest of the day sessions.

Thanks.

- Raj K



and Tom said...

I'm going to pile on here -- I had three more questions about transparent data encryption from the session yesterday.

for all of the technical details:
</code> http://docs.oracle.com/cd/B19306_01/network.102/b14268/asotrans.htm#sthref142


a) How is it licensed.  It is an option available with Oracle Advanced Security.

http://docs.oracle.com/cd/B19306_01/license.102/b14199/options.htm#sthref37

b) can the primary key be "transparently encrypted".  Answer is yes.



ops$tkyte@ORA10GR2> create table t
  2  ( ssn   varchar2(20) encrypt no salt,
  3    data  varchar2(20)
  4  )
  5  /
 
Table created.
 
ops$tkyte@ORA10GR2>
ops$tkyte@ORA10GR2> create unique index t_idx on t(ssn);
 
Index created.
 
ops$tkyte@ORA10GR2>
ops$tkyte@ORA10GR2> alter table t add constraint t_pk primary key(ssn)
  2  /
 
Table altered.
 
ops$tkyte@ORA10GR2>
ops$tkyte@ORA10GR2> insert into t values ( 'abc', 'x' );
 
1 row created.
 
ops$tkyte@ORA10GR2> insert into t values ( 'abc', 'x' );
insert into t values ( 'abc', 'x' )
*
ERROR at line 1:
ORA-00001: unique constraint (OPS$TKYTE.T_PK) violated
 
http://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_7002.htm#CEGDFHBD
lists the restrictions (no foreign keys may be encrypted, so... Now, the question yesterday was, "if SSN would be a natural key, could you encrypt it", the answer is "yes", with a caveat - it could not then be a foreign key:

ops$tkyte@ORA10GR2> create table t2 ( ssn references t );
create table t2 ( ssn references t )
                  *
ERROR at line 1:
ORA-28335: referenced or referencing FK constraint column cannot be encrypted


So, in this case, we would use a surrogate key, not SSN.


c) can we use different encryption schemes?  Yes, we can:
http://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_7002.htm#CEGDFHBD

shows we can use:
3DES168, AES128, AES192, and AES256.



Now, for your questions.  

"who"
http://docs.oracle.com/cd/B19306_01/network.102/b14268/asotrans.htm#sthref142 <code>
shows the architecture. But basically, the server processes do this - so to the end user, it is totally transparent that the data was stored encrypted.


"how" -- see that architecture, I think that chapter will answer all.


"what version", this is 10g Release 2 and above ONLY.


Thanks for the feedback, appreciate that.

Rating

  (42 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

looks like good to go with enterprise edition

John Hurley, July 22, 2005 - 1:21 pm UTC

Yes I did follow the link you cited on licensing. But it doesn't appear to mention the Transparent encryption specifically.

It looks like if you run EE on 10g Release 2 you have it already otherwise if running SE you have to license advanced security if I understand correctly.

Tom Kyte
July 22, 2005 - 1:54 pm UTC


The sentence pointed says:

...
Oracle Advanced Security provides transparent data encryption of data stored in the database....


without ASO, transparent data encryption support is not there.

ASO is an option, that is how I read it.

Amarjyoti Dewri, August 06, 2005 - 3:16 pm UTC

Hi Tom,

Regarding TDE, I wanted to know how to setup multiple databases (Oracle10g Rel2) using different locations of the DIRECTORY path in WALLET_LCOATION parameter of SQLNET.ORA file.

As I could understand, the default location is $ORACLE_BASE/admin/$ORACLE_SID/wallet. This works fine as long as all the databases are using the default location. But say for security reasons I want the wallet to be placed in different locations for different databases. How do we do that?

Thanks and Regards

Tom Kyte
August 06, 2005 - 5:13 pm UTC

I haven't tested this, but -- the default location of the sqlnet.ora which may contain the path to the wallet may be overridden via the TNS_ADMIN environment variable.

But, why "for security reasons", what would you be gaining by putting it elsewhere? Given that any sysdba for database 1 would be able to figure out where it is for database 2 immediately any how?

What do you think you would be buying by doing this?

Thanks for he response

Amarjyoti Dewri, August 07, 2005 - 12:15 am UTC

Hi Tom,

Thanks for replying. Actually I was thinking about the example if disks are stolen. The chances of getting the disks containing the datafiles as well as the wallet together might get reduced. And if we have different wallets in different locations for different databases, the chanches are reduced further.

Location of sqlnet.ora can be controled by TNS_ADMIN parameter. But the parameters WALLET_LOCATION/ENCRYPTION_WALLET_LOCATION in sqlnet.ora is one per ORACLE_HOME. Can we have multiple directories mentioned in parameter ENCRYPTION_WALLET_LOCATION/WALLET_LOCATION?

This is what I understood till now. By default database searches for ewallet.p12 file in path mentioned by ENCRYPTION_WALLET_LOCATION, if it's not mentioned then in WALLET_LOCATION, else picks it from the default database location which is $ORACLE_BASE/admin/$ORACLE_SID/wallet.

If we mention the directory path in ENCRYPTION_WALLET_LOCATION, then all databases will be accessing the same ewallet.p12 file which can have one master key common for all. This is what I want to avoid, I do not want all databases to have same master key, or the same ewallet.12 file (as suggested by the manual, have different wallets for different databases).

One work around that I could think of and did a test while writing this review, was by creating a soft link. Create ewallet.p12 files for databases in their respective default database locations with different master keys for each database. Then move them to a common location or different location and rename them to ewallet$ORACLE_SID.p12. Then create soft links for these files with name ewallet.p12 in the default database location. This is working...

Thanks and Regards
Amar

Tom Kyte
August 07, 2005 - 9:35 am UTC

all you need is

a) wallets over here
b) disks over there

you need "two" not "N"


why do you say that about the sqlnet.ora? If you have tns_admin pointing to /d01 for instance 1 and /d02 for instance 2, that was my point, if each instance has their own sqlnet.ora.....


but I don't see the point, even if they steal your datafiles and your wallet, how are they going to open it?

And how does putting wallets all over the place help? you need two locations in your concept.

About Stealing the Wallet

Arup Nanda, August 08, 2005 - 1:12 am UTC

I apologize for the uninvited intrusion to Tom's forum; but I thought I may have some suggestions for the reader who asked this question about the datafiles and/or wallets being stolen; and perhaps try to put his mind at ease.

First, here is how the TDE mechanism functions. The wallet is not opened by default when the database instance starts. The DBA opens it by issuing "alter system set encryption wallet open authenticated by <password>". It requires a password to be opened, otherwise the masterkey is pretty much useless.

Second, the masterkey is stored inside the wallet in an encrypted manner; whihc means even if the attacker steals it, he will not be able to use it much. So it does not hurt to have the same masterkey for all databases. In other words the risk is the same, not different.

Now, let's discus scenarios. Scenario# 1 - The attacker steals the disks, or the BCV copy of the disks or the backup tape files. He then creates an instance on a test machine, starts up the instance, does an incomplete recovry from the tapes (or disks). Can he read the encrypted columns? Nope. After opening the instance and before accessing the columns, he has to open the wallet, which requires a password. Yes, he could use brute force to guess the password; but that makes any user security inside Oracle vulnerable, not just this one; you could even brute force SYS password, at least in thoery.

One suggestion I would have for TDE team is to enable profiles like they are for users, which enforce better passwords and lock accounts after a while.

Scenario#2: The database is running and the the attacker has gained access to the database. This scenario has one difference - the wallet is open. Sure the attacker can read the .p12 file, but for what? He can't read the encrypted value and he has no idea how the value has been encrypted.

Remember, TDE is not for selective en/decryption of data, i.e. decryption is made available to some users only. All users who have access to the table (or the row, if under VPD) will be given access to the decrypted value. The attacker does not need to know the masterkey to decrypt them. If you must protect the column from a type of users, e.g. encrypted SALARY should be protected from all non-HR users, merely create a column-sensitive VPD policy on the table which will prevent showing the columns based on userid. You could go creative in defining the policy in any way you want - based on application contexts, based on IP addresses, the list goes on.

I happen to think that TDE is a very, very good feature in Oracle 10gR2, and a long-overdue one. It does have its shortcomings such as inbality to define function based indexes, being part of partition keys, etc.; but it can be used to acheive some quick compliance with the myriads of regulations you may have to deal with now.

Hope this helps.

Tom Kyte
August 08, 2005 - 7:45 am UTC

A much more detailed way of saying:

...
but I don't see the point, even if they steal your datafiles and your wallet,
how are they going to open it?
.....

thanks for the input Arup, appreciate it.

Amarjyoti Dewri, August 08, 2005 - 1:31 am UTC

Hi Tom,

Thanks for the reply.

>> why do you say that about the sqlnet.ora? If you have tns_admin
>> pointing to /d01
>> for instance 1 and /d02 for instance 2, that was my point, if each
>> instance has
>> their own sqlnet.ora.....

Oh, you were saying about RAC environment, where every instance will have its own ORACLE_HOME and sqlnet.ora. I think I got overexcited by the feature and did not think in that aspect. I was talking about a single server with single Oracle10g Rel2 ORACLE_HOME hosting multiple databases.

>>but I don't see the point, even if they steal your datafiles and your
>>wallet,
>>how are they going to open it?

Yes, you are right; I was under the impression that doing a decrypt after generating new master key will open the encrypted column. But it does not work that way. It throws error; ORA-28362: master key not found, confirming that you will still need the original master key (generating new wallet does not work unless generated using same master key) to decrypt.

>>And how does putting wallets all over the place help? You need two
>>locations in your concept

Here also I was talking about single ORACLE_HOME for multiple databases. But its clear now, unless master key is available data cannot be accessed.

One more thing I was doubtful about was that, when we open the wallet its open for all users, I could not find any thing saying open only for one particular user. So till the wallet is open anyone having the access to the table is able to see the data from the encrypted column. Only closing of wallet disables access to the encrypted column. So itÂ’s something like a bank manager has the master key to open the wallet, and without his permission no one access the encrypted data. But when he opens the wallet, others can see the data as well...

To prevent this wouldn't we need the same FGA or view where data was restricted user wise or role wise.

So I was wondering where actually is TDE useful?

Thanks and Regards
Amar


Tom Kyte
August 08, 2005 - 7:47 am UTC

NO, i was not talking about rac.

I was talking about:

export TNS_ADMIN = /d01
export ORACLE_SID = foo
sqlplus "/ as sysdba" <<EOF
startup
EOF

export TNS_ADMIN = /d02
export ORACLE_SID = bar
sqlplus "/ as sysdba" <<EOF
startup
EOF


foo and bar have different "?/network/admin" directories now because they have different tns_admin's -- but this would be confusing at best and not achieve anything "extra security wise"


I think Arup's comments above answers the last question.



Amarjyoti Dewri, August 08, 2005 - 1:41 am UTC

Hi Again,

Sorry I missed Arup's answer while I was typing. What I understand is that you need to combine both VPD and TDE together to provide a better and robust security at both application level and datalevel. Am I correct?

So you can encrypt the data with Transparent Data Encryption and prevent access to confidential data while wallet is open using Vitual Private Database where only specific users can access data, and that too will not be available when you close the wallet.

Thanks and Regards

Tom Kyte
August 08, 2005 - 7:49 am UTC

TDE is to protect data at rest, someone steals your backup - so what, they cannot read the sensitive information.

VPD (FGAC) is to protect data from being accessed. From end users, from people who are supposed to see some of the data but perhaps not all of the data.




For Arup Nanda

Amarjyoti Dewri, August 08, 2005 - 2:43 am UTC

<Quote>First, here is how the TDE mechanism functions. The wallet is not opened by
default when the database instance starts. The DBA opens it by issuing "alter
system set encryption wallet open authenticated by <password>". It requires a
password to be opened, otherwise the masterkey is pretty much useless.
</Quote>

But Arup, there is an option available for keeping the wallet open all the time to be used for Single Sign On Applications (only scenario where it should have autologin as suggested by Oracle Manuals). Turing on autologin in wallet manager enables the wallet to be open all the time. It create a cwallet.sso file for this. I did some testing on this which can be accessed here at the bottom of this page </code> http://adewri.modblog.com/?show=custom&page=2658 <code>So you do not have to reopen the wallet when the instance is restarted. What precautions should be taken in such a scenario...

Thanks and Regards

Tom Kyte
August 08, 2005 - 7:59 am UTC

do not use that would be the simple response.


where does it say in order to use SSO with TDE you must do this for the DATABASE? I believe that (setting auto-login) is in regards to the user wallet, with the user credentials.

Thanks a lot

Amarjyoti Dewri, August 08, 2005 - 8:33 am UTC

Hi Tom,

[quote]I believe that (setting auto-login) is in regards to the user wallet, with the user credentials. [/quote]

Yes, you are right , it is in regards to user credentials (you miss important points when you don't read carefully. I will be careful next time).

I would like to thank you and Arup Nanda for clarifying all my doubts.

Thanks and Regards

Its all nice and beautiful but....

Dominic Dougherty, September 19, 2005 - 8:53 pm UTC

Great website, Very helpful, Oracle Rocks!!
One thing regarding TDE in 10gr2, I know it encyrpt data and it is transparent but does it do access control. It does separate dba from Security Officers But will it be able to restrict decryption to specific users.

Tom Kyte
September 19, 2005 - 11:45 pm UTC

it is *not* access control -- it is protection from THEFT of data.

vpd (dbms_rls) is for access control
grant is for access control

abotu storing keys in memory

Antonio, September 20, 2005 - 5:17 am UTC

Tom, do you know if the database engine is storing the decrypting-keys in memory locked? My reasoning is that if they are not forcefully locked (mlock() syscall on unix), they could be written out to swap space and then recovered from there.

Tom Kyte
September 20, 2005 - 10:10 am UTC

I don't know that low level detail - you'd have to ask support to track that down via development.

Operational Procedures

VA, September 20, 2005 - 3:10 pm UTC

<quote>
but I don't see the point, even if they steal your datafiles and your wallet, how are they going to open it?
</quote>

If the disks/tapes containing both the datafiles and the wallet are stolen, you still need to "alter system set encryption wallet open authenticated by <password>" to access the data.

But, in many environments, the database open is a part of the server startup procedure and this "command" above would be in some home-grown "dbstart" script or some GUI/OEM equivalent.

So, if the "attackers" gets hold of all the disks involved, they could (theoretically) access the data, couldnt they?

Thanks


Tom Kyte
September 21, 2005 - 1:27 pm UTC

well, that sounds like a "in many environments" issue doesn't it?

Sure, if you put everything there -- well, what can we do?

Wallet could be on another system even as well.

But yes, you hard code passwords and - well, what can we do?

A reader, September 21, 2005 - 12:37 am UTC

To VA from New Jersey, USA

>>So, if the "attackers" gets hold of all the disks involved, >>they could (theoretically) access the data, couldnt they?

Then you got bigger problems then security :-)

Tom Kyte
September 21, 2005 - 6:56 pm UTC

not really, it happens all of the time.

The day I presented at OOW this year, talking about transparent data encryption, this was in the news:

</code> http://www.sfgate.com/cgi-bin/article.cgi?file=/c/a/2005/09/20/BAG1LEQEM41.DTL&type=printable <code>

I used it as one of my slides, very timely.

Some TDE notes

Adam Musch, October 26, 2005 - 3:45 pm UTC

One thing missing from the documentation (or perhaps implied that I missed) was that after setting ENCRYPTION_WALLET_LOCATION or ENCRYPTION_LOCATION in sqlnet.ora, one must shutdown/startup the database in order to create a wallet for TDE.

As a developer/DBA in the banking world, TDE is a godsend compared to hardware/software encryption. Compared to our systems running Oracle 9.2 with a java front-end, using a hardware/software solution with views and INSTEAD-OF triggers imposes about a 50% performance penalty, requiring twice as much CPU to comply with industry requirements.

The same data and front end running with TDE (so obviously 10gr2) not only had no penalty, but had approximately 4 times better throughput than the original 9.2 system without encryption!

I think it'll be a pretty easy sell to say:
Well, we can double our hardware and database licensure cost, or:
We can license the Advanced Security option for 10k/cpu list.

One thing about TDE though: It doesn't support CLOB/BLOB columns. However, digging through the 10gr2 documentation (specifically the DBMS_CRYPTO spec) shows that one can use DBMS_CRYPTO to encrypt LOB columns with AES128/192/256 and 3DES, all of which are supported natively with TDE.

Why didn't they make LOB columns natively supported through TDE? It looks like all the pieces are there, but they just didn't stitch them all together. I've no desire to manually attempt to perform key management and use DBMS_CRYPTO to encrypt those columns.

Do we think/know/suspect that support for LOB columns with TDE will be forthcoming, in either a major or a minor release?

Thanks!

Tom Kyte
October 27, 2005 - 4:46 am UTC

I'll ask and see if I can find out. Problem with lobs is you have piecewise access to them - you can dbms_lob.write bytes 100 - 200. They are like files - not like a "variable". There are complexities involved here you haven't thought about that make doing it to lobs really really hard.

TDE and Relational Integrity

Adam Musch, October 31, 2005 - 12:17 pm UTC

Any word on whether TDE will support RI in a future release?

Currently, each table appears to have its own initialization vector for encryption (salt), meaning a value encrypted in one table differs from a value encrypted in another table. Salt can also be added at the column level, so that repeating values in the column have different ciphertexts. However, if specifying the IDENTIFIED BY clause of the ENCRYPT command as well as the NO SALT clause, the underlying ciphertext is identical both across tables and across columns. (Thanks to markmal at dizwell for pointing this out.)

Example:

drop table t_owners;
drop table t_tables;

create table t_owners as select * from all_users;

create table t_tables as select * from all_tables;

alter table t_owners
add constraint pk_t_owners primary key (username)
using index;

alter table t_tables
add constraint pk_t_tables primary key (owner, table_name)
using index;

alter table t_owners
modify (username encrypt identified by "encryption_key" no salt);

alter table t_tables
modify (owner encrypt identified by "encryption_key" no salt);

select dbms_rowid.rowid_row_number(rowid) as rn,
DBMS_ROWID.ROWID_BLOCK_NUMBER(rowid) as bn
from t_owners where username = 'WMSYS' AND ROWNUM = 1;

SELECT dbms_rowid.rowid_row_number(rowid) as rn,
DBMS_ROWID.ROWID_BLOCK_NUMBER(rowid) as bn
FROM T_TABLES WHERE OWNER = 'WMSYS' AND ROWNUM = 1;

After dumping the two blocks, here was the pertinent data:

Block dump of t_owners:
tab 0, row 25, @0x15e0
tl: 51 fb: --H-FL-- lb: 0x2 cc: 3
col 0: [36]
54 7d d1 5b e2 12 16 83 86 6c c5 bb 1a 81 52 f7 12 cf f2 26 a7 6b cd aa 53
53 57 54 24 6f a6 04 a5 a1 76 fb
col 1: [ 2] c1 1a
col 2: [ 7] 78 69 0a 18 08 22 06

Block dump of t_tables
tab 0, row 8, @0x16c5
tl: 245 fb: --H-FL-- lb: 0x2 cc: 49
col 0: [36]
54 7d d1 5b e2 12 16 83 86 6c c5 bb 1a 81 52 f7 12 cf f2 26 a7 6b cd aa 53
53 57 54 24 6f a6 04 a5 a1 76 fb
col 1: [16] 57 4d 24 4e 45 58 54 56 45 52 5f 54 41 42 4c 45
col 2: [ 6] 53 59 53 41 55 58
col 3: *NULL*
...

Note that the VARCHAR2(30) column is now a 36 byte column (which is expected for encryption to round up to a multiple of 8), and that the ciphertext for the two is identical.

So it certainly appears that TDE could support RI between columns encrypted with the same value of the IDENTIFIED BY clause.

Any idea why it wasn't implemented? Or (oh, please, oh, please) it'll be resolved with a patch?

Tom Kyte
November 01, 2005 - 5:09 am UTC

I don't know "why", I can comment that if the primary key needs to be encrypted, I would think 500 times before using that as my primary key.

Doesn't seem like it would be the right column to have exposed as the primary key at that point.

TDE and Legacy Systems

Adam Musch, November 01, 2005 - 8:58 am UTC

Tom:

The issue is not with a new design; it's with adapting TDE with existing systems where new industry requirements (such as HIPAA and Visa/Mastercard's PCI) require database encryption. The natural key (let's say, an account number within a banking system) now must be encrypted. The requirement didn't exist when the systems were originally designed, and lots of internal developers and vendors are scrambling to find a way to comply.

I do agree with you that now, it would be obvious to use a surrogate/synthetic key in this case. However, right now that would involve reimplementing the key structures for lots and lots of tables, which means lots and lots of code rewrite.

I'll probably end up using triggers to enforce RI -- it's not good, but it is better than not having the RI at all. The tables in question are fairly static; there's not lots of insert/delete activity on the tables in question, and the keys are already immutable.

I will watch out for when TDE does support RI where the column "public" keys are identical and no salt is added at the row level.

Thanks for all of your assistance, both on this issue and all the other issues you help the community with.

Tom Kyte
November 01, 2005 - 11:02 am UTC

becareful - you show me your code for RI in triggers and I'll tell you why it is buggy :) (high probability - give it a go, I'll look at them if you post them and tell you why they won't work - or verify that you got it - it is very very tricky and involves typically the use of "lock table", like we do with unindexed foreign keys).


Best place to get your ideas into the product however is metalink - you should file an enhancement requestion.

Encrypting all columns

Partha, November 16, 2005 - 6:53 am UTC

We are developing a HTMLDB application that can be accessed through the internet. All the data is very confidential. The application / data itself is quite small with around 30 tables and 100 columns in all and around 10,000 records. One of the suggestions was to encrypt all the user visible columns in all the tables (ofcourse not the clobs / foriegn keys etc). But currently we are using the standard edition one. Is upgrading to EE and buying a ASO for this a viable option? That would also mean making all the columns 'encypt'. Is there any other option ?

Tom Kyte
November 16, 2005 - 5:09 pm UTC

who are you trying to protect the data from?

The transparent in transparent data encryption means something - the data is transparently encrypted for viewing by anyone that is allowed to see it.

TDE is useful to protect against data theft (someone stealing your database).
TDE is *not access control*

You sound more like "access control", not encryption.


So, who are you trying to protect the data from?

Followup to review "Some TDE notes October 26, 2005"

A reader, December 22, 2005 - 10:48 am UTC

In your follow up to the review titled "Some TDE notes October 26, 2005" you mentioned that you will ask about when Oracle 10g will support encryption of LOBs. I was wondering if you got an answer.
If not, is there a work around, while still using TDE, to encrypt LOBs?

Thanks

Tom Kyte
December 22, 2005 - 11:33 am UTC

there not plans to do so currently - no. you can use the procedural API's to do so, but obviously that is not "transparent"

Startup scripts?

Ben, March 13, 2006 - 3:34 pm UTC

Tom,
As I understand it, once the wallet is created, it must be explicitly opened every time the database starts up to grant users access to the data in the encrypted column.

How do we deal with that in the context of automated RMAN scripts, run in cron jobs, that shut down the database, mount it for a full backup, and open the database? If we add a command in there to open the wallet, including the password, doesn't that defeat the purpose of encrypting the master key?

Tom Kyte
March 14, 2006 - 9:55 am UTC

You can make the wallet "releasable to the database automatically" using a director (LDAP repository).

Yes, if you hard code a password in a script, that defeats the purpose.

Unattended startup and shutdowns are something I've written about many hundreds of times in the past - I never do them. They are fraught with issues. If you have a serious database, with serious data (so serious you would consider encrypting it), using enterprise edition (which you must be) you would never be shutting down for a backup. You would be in archive log mode, you would be using hot backups.

Applying TDE to big table

Mihajlo, April 06, 2006 - 2:41 pm UTC

Hi Tom,

I was testing encryption (TDE) of certain columns in a table which contains 27 million records.
Adding TDE encryption to VARCHAR2 column took 1 hour and 30 minutes. Adding TDE encryption to two NUMBER columns at the same time took less than 5 hours.

Can you explain me why it takes so much time for encrypting the data? I understand the data needs to be converted and than saved in the table, but is there anything which can help me to reduce this time.


Tom Kyte
April 07, 2006 - 8:20 pm UTC

you need to basically rewrite the entire table when you are doing this (logged operation as well). Also - encryption is reputed to be somewhat CPU intensive.

Once added, it is unlikely you will notice the difference in most cases - sure, there are extreme cases that you could (on the decrypt) - but it is unlikely.

RE: Applying TDE to big table

Mihajlo, April 10, 2006 - 3:00 pm UTC

What do you mean with "rewrite the entire table"?

Do you think all columns data need to be modified, or only for the columns which are encrypted with TDE?

Thanks

Tom Kyte
April 11, 2006 - 11:41 am UTC

the raw encrypted data is bigger than the original data. The entire table is pretty much rewritten, just like an

update t set x = x || 'x';

would - even if T has 100 columns - you are modifying every row in every block, table is rewritten in effect.

dbms_crypto

A reader, September 17, 2006 - 4:06 pm UTC

Is dbms_crypto available to ORacle 10gR2 Standard Edition?

Tom Kyte
September 18, 2006 - 1:03 am UTC

yes

DataGuard and TDE

Tino, March 05, 2007 - 10:16 am UTC

Will TDE work with DataGuard due to having the master key in a wallet outside of the database? How do we go about keeping the master keys in-synch (primary and standby sites)?
Tom Kyte
March 05, 2007 - 2:12 pm UTC

only using a physical Standby Database, the wallet must be copied manually.

Also , you need to specify the ENCRYPTION_WALLET_LOCATION in the sqlnet.ora file on the standby database.

DataGuard and TDE

Tino, March 05, 2007 - 5:14 pm UTC

So based on your response, I cannot implement TDE if I am configuring a Logical Standby database? Is that correct?
Tom Kyte
March 05, 2007 - 8:48 pm UTC

correct - the generation of the logical change record would sort of break the entire thing.

Good Memory or Good Search Engine

Tino, March 06, 2007 - 2:29 am UTC

You are pretty good, I overlooked it in the documentation under "Unsupported Data Types in a Logical Standby Database".

CTAS with TDE

Skip, March 15, 2007 - 10:36 am UTC

Is there a way around the CTAS issue with TDE? I understand that TDE is for data theft, or if someone steals the database. But if I have a production table that has say account number and it is encrypted, and my user does a CTAS as select * from the production table, the resulting table is not encrypted and therefore I now have unencrypted data in the database ready to be stolen. I am not able to prevent users from creating their own tables as it is integral to their daily work. How can I ensure that any tables they create are encrypted, DDL trigger?

Tom Kyte
March 15, 2007 - 12:32 pm UTC

seems to me you have a serious conflict between your stated needs (encryption to prevent theft) and your application (users can willy nilly copy the data all over the place at will)

you could create a DDL trigger that looks at the table being created and the table it is being created from and errors out of any of the columns are encrypted...


archive logs?

dulu, June 01, 2007 - 10:43 am UTC

Hi Tom

Please can you confirm that if you have all your data encrypted using TDE, then also the backups and archive logs are also encrypted?

Thanks
Tom Kyte
June 01, 2007 - 1:16 pm UTC

you would not have all of your data encrypted with TDE, you would selectively apply this only to columns that MUST be protected.

and those columns would be encrypted in the backups and everywhere.

a question on TDE?

Sona, July 12, 2007 - 7:44 pm UTC

Hi Tom,

1.How do we decrypt the columns encrypted with TDE, when the wallet password is forgotten or unknown since it was created by someone else?
2.How do we reset the wallet password?
3.I understand that expdp is the option to export table encrypted with TDE and we need the master key to decrypt the encrypted data. How do we import the table in another database in different host, which has a different wallet in use/open?

Thanks,
Sona

Re: Questions on TDE

sona, July 13, 2007 - 7:32 pm UTC

Hi Tom,

Thanks much for the answers.

1. Regarding 'RESET' -- I just wanted to know how do we change the password of the wallet if it has been compromised in any ways?

For eg: We create the wallet using the following command.
alter system set encryption key authenticated by "passwordwal";
How do I change password to 'newpassword'?

2. Regarding Backup/Recovery:
I understand that we need to backup wallet along with the database backup for ensuring restore and recovery of the table encrypted with TDE.
Suppose we want to recover the table by using RMAN restore/recover tablespace, in another database which already has a wallet in use.
How do we use the wallet of the Database1 restored from backup to open the table restored/recovered in Database2?

3. How does TDE work with standby databases? Do we need to have wallets in standby hosts too?

Thanks,
Sona
Tom Kyte
July 13, 2007 - 7:55 pm UTC

1) check out that chapter i pointed to, all covered there.

2) see number 1
3) same (yes, you would in order to open the database)

TDE Performance Increases?

Tom, September 29, 2007 - 1:59 am UTC

Hi Tom - Great site, great name, and looking forward to OOW. In trying to gauge the impact of installing TDE, we've run some tests, and find that btree indexes seem considerably faster (200 million row test table, 4 times faster, with less CPU microseconds). Is that expected due to the increased uniqueness of the indexed value? For example, less ids look like 1234%? Or am I getting odd results from a flawed test run?

Hope this isn't a repeat question (or should have been flagged as a new question), but I've not been able to track down much regarding TDE impacts. We're seeing slower FTS and partition scans (more blocks, but that's expected), and faster index range scans with less CPU ... completely unexpected!

Thanks for this site, and all your expertise!
Tom
Tom Kyte
October 03, 2007 - 1:40 pm UTC

TDE cannot do index range scans - what did your test actually test?

More TDE Performance Info

Tom, October 04, 2007 - 11:21 am UTC

Well, not an index range scan in the sense of "ID like '12345%', but index lookups (ie, id='123451234'). I always see the explain plan call that lookup an index range scan, so guess I picked up some poor terminology.

When we look at our encrypted index, we see that it's roughly twice as large after encryption, (clustering factor is about the same on both, data was not sorted) and expected to see some degradation due to the increased IO. However, even with flushing buffer_cache, and shared pool, running our tests with new sets of ids, running encrypted query test first or running them after it's equivilant unencrypted counterpart, we still see less logical reads, less physical reads, less CPU and less elapsed time.

Here's some stats from our runs, do you have any tests that show index lookups are actually faster? Our indexed column is varchar2, and the values are not random.

Encrypted Test:

------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Pstart| Pstop |
------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1938K| 123M| 3285 (6)| | |
| 1 | HASH GROUP BY | | 1938K| 123M| 3285 (6)| | |
| 2 | PARTITION RANGE ALL | | 1938K| 123M| 3110 (1)| 1 | 5 |
| 3 | TABLE ACCESS BY LOCAL INDEX ROWID| TEST_DETAIL2 | 1938K| 123M| 3110 (1)| 1 | 5 |
|* 4 | INDEX RANGE SCAN | TEST_DETAIL2_NDX7 | 775K| | 3106 (1)| 1 | 5 |
------------------------------------------------------------------------------------------------------------------

select ENC_TEST_ID, sum(AMOUNT) from test_detail2 where ENC_TEST_ID = '1234567890123456789' group by ENC_TEST_ID;
Elapsed: 00:00:00.39

Statistics
----------------------------------------------------------
1064 recursive calls
0 db block gets
285 consistent gets
62 physical reads
0 redo size
773 bytes sent via SQL*Net to client
488 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
13 sorts (memory)
0 sorts (disk)
1 rows processed
From v$sqlstats:
CPU_TIME: 40325
ELAPSED_TIME: 376289
IO_WAIT_TM: 340019



Unencrypted Test:

select TEST_ID, sum(AMOUNT) from test_detail where TEST_ID = '1234567890123456789' group by TEST_ID;
Elapsed: 00:00:02.90

Execution Plan
----------------------------------------------------------

-----------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Pstart| Pstop |
-----------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 2 | 66 | 8 (13)| | |
| 1 | HASH GROUP BY | | 2 | 66 | 8 (13)| | |
| 2 | PARTITION RANGE ALL | | 2 | 66 | 7 (0)| 1 | 5 |
| 3 | TABLE ACCESS BY LOCAL INDEX ROWID| TEST_DETAIL | 2 | 66 | 7 (0)| 1 | 5 |
|* 4 | INDEX RANGE SCAN | TEST_DETAIL_NDX2 | 2 | | 6 (0)| 1 | 5 |
-----------------------------------------------------------------------------------------------------------------

Statistics
----------------------------------------------------------
1034 recursive calls
0 db block gets
2974 consistent gets
191 physical reads
0 redo size
779 bytes sent via SQL*Net to client
488 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
12 sorts (memory)
0 sorts (disk)
1 rows processed

From v$sqlstats:

CPU_TIME: 450156
ELAPSED_TIME: 2827453
IO_WAIT_TM: 2360136

TDE Unexpected Performance Explained?

Tom, October 05, 2007 - 10:56 am UTC

Hi Tom - We found the section of sql that was causing the discrepancy in our timings in our 10046 trace file. For an unencrypted query, Oracle ran the following, accounting for the majority of our overhead. On the unencrypted side, this took zero seconds ...

SELECT /* OPT_DYN_SAMP */ /*+ ALL_ROWS IGNORE_WHERE_CLAUSE
NO_PARALLEL(SAMPLESUB) opt_param('parallel_execution_enabled', 'false')
NO_PARALLEL_INDEX(SAMPLESUB) NO_SQL_TUNE */ NVL(SUM(C1),:"SYS_B_0"),
NVL(SUM(C2),:"SYS_B_1")
FROM
(SELECT /*+ IGNORE_WHERE_CLAUSE NO_PARALLEL("TEST_DETAIL")
FULL("TEST_DETAIL") NO_PARALLEL_INDEX("TEST_DETAIL") */
:"SYS_B_2" AS C1, CASE WHEN "TEST_DETAIL"."TEST_ID"=
:"SYS_B_3" THEN :"SYS_B_4" ELSE :"SYS_B_5" END AS C2 FROM
"TEST_DETAIL" SAMPLE BLOCK (:"SYS_B_6" , :"SYS_B_7") SEED (:"SYS_B_8")
"TEST_DETAIL") SAMPLESUB


call count cpu elapsed disk query current rows
------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.01 0.00 0 0 0 0
Fetch 1 0.46 2.13 148 2833 0 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 3 0.47 2.14 148 2833 0 1

Our DBAs are still looking into the difference, but at least now we can safely say that TDE is not boosting our performance...
Thanks!

TDE column with Replication is not supported

A reader, January 18, 2008 - 1:08 pm UTC

Tom,
We replicate a table that we turned TDE on. We use Shareplex replication to replicate this table from Production to Reporting. Shareplex does not support TDE and in my research -I have not come across any replication -even Streams support for TDE. can you please comment..
Regards
Praveen
Tom Kyte
January 19, 2008 - 10:43 pm UTC

http://docs.oracle.com/docs/cd/B19306_01/network.102/b14268/asotrans.htm#BABEEEIH

basically, in 10gr2, if you replicate encrypted data, it'll be in the replication queues "not encrypted"



Let me say this, if the data is sensitive enough to be encrypted - I cannot imagine you replicating it, what be this sort of data? Once you replicate it, you lose control of it.

Here's that URL again

Stew Ashton, January 20, 2008 - 3:16 am UTC


http://download.oracle.com/docs/cd/B19306_01/network.102/b14268/asotrans.htm##BBABEEEIH
And I turned bold off for the rest of this page. Gosh, think how much less fun it would be if this bug were fixed !
Tom Kyte
January 20, 2008 - 7:59 am UTC

I deem it not a bug, I need the # b and # B things to function, so be it.

Re : you deem it not a bug

Stew Ashton, January 20, 2008 - 5:03 pm UTC


Sorry about using the "b" word, I had "b"s on the brain :)

Granted, you need # b to function, but do you need it to function inside a URL?

Wouldn't it be nice if all the URLs functioned too?

The code could be changed to either ignore tag formatting within a URL, or change "# b" to "## bb" within a URL, which is what I did by hand. Might not that keep you happy and help out the folks who are going to click on these links?

My two ## worth :)

TDE with PKI key pairs certificate

A reader, June 24, 2008 - 4:48 pm UTC

Hi Tom,

We want to implement TDE with PKI key pairs currently being used in the company. But when I generated the user certificate, it has
KeyUsage=Digital Signature, Key Encipherment

As Oracle suggests that the KeyUsage field should be marked for encryption or left blank.

Per Oracle doc:
===========================================
To use transparent data encryption with PKI key pairs, the issuing certificate authority must be able to issue X.509v3 certificates with the key usage field marked for encryption.
==============
Remark: for a certificate to work this way, the certificate authority must not use any Key Usage specification to sign the certificate. If you have problems seeing the certificate in V$WALLET, create a new certificate without any Key Usage specifications.
To verify the certificate, export it from the wallet using OWM and give the file an extension .crt.

For example: mycertificate.crt,
then double-click the certificate, or right-click the certificate,
then click Open , the Certificate dialog box appears,
go to [Details] and Show: [Extensions Only],
this must only include: [ Basic Constraints, Netscape Comment, Subject Key Identifier, Authority Key Identifier ]
and not [Key Usage] .

I am a little confused as it indicates KeyUsage "marked for encryption" in first paragraph, while the second paragraph says "certificate without any Key Usage specifications".

I talked to our security staff, they do not know how to have the certificate specified by Oracle generated. The certificates our website generated are with KeyUsage specified. Do you have any insights on this?

Thanks,

Issue with Oracle 8i talking to Oracle 10g with Encryption

Anne, February 06, 2009 - 2:47 pm UTC

Hi Tom,

I hope my question pertains to this topic.

We have an Oracle 8i database that talks to a remote Oracle 10g database using a database link. 

When querying the remotetable from our 8i database, we now get this error :

SQL> select version from v$instance;

VERSION
-----------------
8.1.5.0.0

SQL> desc remote_table@remotedblink;
ERROR:
ORA-12650: No common encryption or data integrity algorithm


This has been working fine until early February 2009, when the remote site turned on  Enforced Encryption.

The tech person at the remote site tells me that Oracle 8i database in not capable of talking to an Oracle 10g database with encryption turned on.

I googled around and searched Metalink, but did not find any documentation for this.

Please advise. Would appreciate any help.

Thanks,
Anne

Tom Kyte
February 06, 2009 - 4:17 pm UTC

do you have ASO (advanced security) licensed and available on your (very much way out of support) 8.1.5 database?

Issue with Oracle 8i talking to Oracle 10g with Encryption

Anne, February 06, 2009 - 4:28 pm UTC

Hi Tom,

I like your "very much way out of support" comment - I'm surprised that's all you said!!! :-)

No, we do not have ASO (advanced security) licensed and available on the 8.1.5 database.

Thanks,
Anne
Tom Kyte
February 09, 2009 - 5:43 pm UTC

you would need it - they are using ASO, you would need the client support that allows you to talk to an ASO 'protected' database - they are demanding encrypted protocols, you would need to satisfy that.

TDE and Index Range Scan

A reader, July 20, 2011 - 5:59 pm UTC

Is it true that in 11g, TDE allows index range scan if the encryption is done at the tablespace level as opposed to the column level encryption?

Also does TDE work with BLOBs and CLOBs?
Tom Kyte
July 22, 2011 - 1:13 pm UTC

with transparent TABLESPACE encryption - you can have indexes without limitations. It is only with column level encryption that you have some limits on index use with encrypted data.

In the very near future, I'll be talking about this in a webcast:

http://event.on24.com/r.htm?e=328712&s=1&k=D11C69EADAB1A25981D9E53966EE5D15&partnerref=blog1_sec_dbsecmulti

if you are interested.

AUTOLOGIN versus System Trigger wallet open

Paul, July 31, 2012 - 12:51 pm UTC

In a comment way back, you mention that you would not use the autologin wallet feature for databases using encryption, but only use a DBA issued command to open the wallet, at least that's the way I interpreted it.

We are having a discussion of the best methodology to use in our prod systems. For various security reasons, the databases have maintennace done to them on a monthly to quarterly basis. They are restarted as part of this process. DBAs are not involved in this security related efforts (unless something goes wrong, of course :-) )

So the decision was made to use a system trigger at startup time to execute the open. The code is wrapped.

I'm having a hard time trying to understand how someone might think this was more secure than an autologin wallet.

I'd appreciate it if you could take a few moments to consider this and offer your opinion.

Thanks in advance!!!
Tom Kyte
July 31, 2012 - 1:06 pm UTC

no, that wasn't what I meant - using the autologin wallet is valid, especially since you can now make it so that the wallet only works on the machine it was created on (complicates distributed stuff though).

So the decision was made to use a system trigger at startup time to execute the
open. The code is wrapped.


oh, that is bad. all i have to do is steal your database and wallet and I'm in!!!! who cares if the code is wrapped, the code will still execute when I restore your stolen database backup. And further, I'll unwrap your code in about 5 seconds or less.


I'm having a hard time trying to understand how someone might think this was
more secure than an autologin wallet.


ummm, me too - since it utterly defeats all of the work put into this. You might as well not encrypt!!!!!



Thanks for those points

Paul, August 01, 2012 - 3:52 pm UTC

Thats what I was thinking, but I tend to be in the minority, theres a lot of "thats the way we always done it" going on around me.

I'd like to investigate the wallet and host item you mention, I'm thinking that might be a new 11g feature. I did a quick check but dont see it, or at least it didnt seem to show up in the master index. Is there a name for that feature or functionality?

Thanks again very much!!!
Tom Kyte
August 01, 2012 - 4:05 pm UTC

why not just point out to management that

a) we encrypt to protect data at rest
b) if we store the password in the database, we have not accomplished a at all.

period.


http://docs.oracle.com/cd/E11882_01/network.112/e10746/asotrans.htm#sthref86

You can unwrap code?

Matthew McPeak, August 01, 2012 - 6:00 pm UTC

>> And further, I'll unwrap your code in about 5 seconds or less.

I thought this was supposed to be impossible?
Tom Kyte
August 02, 2012 - 9:00 am UTC

no, it isn't.


create or replace package my_types wrapped
a000000
1
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
9
87 d2
X5L4x0os9qgz7eEtYUgHbv+1lSIwg6TZLcusfI5EaeqSlL+EeIwT8GrVei3MnkKpLbZ/QKJb
r0Z70rsqSBetKgFJxQRktAfOHMnp8PLs5V355uD6bogkkYDx18CYhzjEgKuwKLnEUm7hizhn
MHrBwwoagSrWooVq3xG1xHvdMmM4NzKtDzh6XzIUvMTiQ6zp1Z++h8/7eAtsAQ==

/


paste that into http://www.codecrete.net/UnwrapIt/


see:
http://docs.oracle.com/cd/E11882_01/appdev.112/e25519/wrap.htm#LNPLS016


Interesting!

Matthew McPeak, August 02, 2012 - 11:49 am UTC

Very interesting, thank you! It makes one wonder what the point of wrapping is.