Skip to Main Content
  • Questions
  • Move historical data between databases

Breadcrumb

Question and Answer

Connor McDonald

Thanks for the question, Victor.

Asked: August 22, 2018 - 6:28 pm UTC

Last updated: October 03, 2018 - 3:54 am UTC

Version: 11.2

Viewed 1000+ times

You Asked

Hello Tom

See how you could optimize moving records (historical by date) from one table in a production database to another table in another historical database in an automatic way.

Could you support me in Oracle Partition?
It could be used exchange partition?

Thanks for the support.

and Connor said...

Partitioning is definitely going to be your friend here, because with conventional means (insert/delete), whilst it can be fast copy the data over to the new target, the cost of removing the data from the source can be very high, eg

SQL> select count(*) from t;

  COUNT(*)
----------
   3961700

SQL> create table t1 as select * from t where 1=0;

Table created.

SQL> set timing on
SQL> insert /*+ APPEND */ into t1
  2  select * from t;

3961700 rows created.

Elapsed: 00:00:08.60
SQL> commit;

Commit complete.

Elapsed: 00:00:00.00
SQL>
SQL> delete from t;

3961700 rows deleted.

Elapsed: 00:00:29.19


So nearly 4 times more expensive (in this demo) to delete.

Exchange partition is not probably going to help you enormously in terms of efficiency, because the main benefit of exchange partition is that the data does not move, and of course, in this case, you actually *want* to move the data (to another database).

But we've send from above, that inserting data is not particular expensive, so if we can minimal the cost of deletes, then we might have a workable solution. For example, the demo below uses a yearly partition to copy a year of data over, and then truncate at the partition level which is far more efficient than delete.

SQL> drop table t purge;

Table dropped.

SQL>
SQL> create table t
  2  partition by range ( created )
  3  (
  4    partition p1 values less than ( date '2018-01-01' ),
  5    partition p2 values less than ( date '2019-01-01' )
  6  )
  7  as select d.* from dba_objects d,
  8   ( select 1 from dual connect by level <= 50 );

Table created.

SQL>
SQL> set timing on
SQL> insert /*+ APPEND */ into t1
  2  select * from t partition (p1 );

3598300 rows created.

Elapsed: 00:00:09.05
SQL> commit;

Commit complete.

Elapsed: 00:00:00.00
SQL> alter table t truncate partition p1;

Table truncated.

Elapsed: 00:00:00.41


If its hard to copy the data in the first, eg, that insert is going across a network etc, then another option to explore is transportable tablespace. In that instance, you would move a partition of data into a new, empty tablespace, and then detach from the primary and attach to the historical one.

So the workflow would be along the lines of:

- move the partition to be archived into tablespace X using 'alter table move partition '
- make tablespace X read only
- unload the metadata with expdp, eg

expdp hr/hr DIRECTORY=dir1 DUMPFILE=tts.dmp TRANSPORT_TABLESPACES=X TRANSPORT_FULL_CHECK=YES LOGFILE=tts.log

- copy the datafiles for X to the historical database
- import the metadata

impdp hr/hr DIRECTORY=dir1 DUMPFILE=tts.dmp TRANSPORT_TABLESPACES=X TRANSPORT_DATAFILES='user01/data/tbs6.dbf' LOGFILE=tts.log

Rating

  (3 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

Victor Guzman, August 28, 2018 - 3:24 pm UTC

Thanks Tom, your answer helps me a lot.
But I think I did not give you enough information to conclude this answer, the excuses of the case.

What happens that my source table to partition has a blob field that is stored in a tablespace different from the rest of the columns and as I said you have to move historical data to another database that is in another site (geographical area)

What I can notice is that the option of transportable tablespace would help me a lot, the queries would be the following:

It helps me to also partition the blob column?

The following script would be correct to partition the table, there I use TS_BLOB for the column blob type, TS_DATA for the data of the rest of columns and TS_IX for the index. Would it be the right one to allow me to transport tablespace?

CREATE TABLE TBL1
(
  ARCHIVE_NAME VARCHAR2 (35 BYTE) DEFAULT '' NOT NULL,
  SIZE_COUNT NUMBER (10) DEFAULT 0,
  HASH_COD VARCHAR2 (40 BYTE) DEFAULT '',
  STATE_COD VARCHAR2 (2 BYTE) DEFAULT '',
  DATE_ING DATE DEFAULT to_date ('01 / 01/0001 ',' DD / MM / YYYY '),
  DATE_MOD DATE DEFAULT sysdate,
  FILE_IMG BLOB DEFAULT EMPTY_BLOB ()
) TABLESPACE TS_DATA
PARTITION BY RANGE (DATE_ING)
INTERVAL (NUMTOYMINTERVAL (1, 'YEAR'))
(
  PARTITION P2017 VALUES LESS THAN (TO_DATE ('2018-01-01 00:00:00', 'SYYYY-MM-DD HH24: MI: SS', 'NLS_CALENDAR = GREGORIAN'))
    TABLESPACE TS_BLOB
    LOB (FILE_IMG) STORE AS SECUREFILE PL2017 (
      TABLESPACE TS_BLOB_D2017),
  PARTITION P2018 VALUES LESS THAN (TO_DATE ('2019-01-01 00:00:00', 'SYYYY-MM-DD HH24: MI: SS', 'NLS_CALENDAR = GREGORIAN'))
    TABLESPACE TS_BLOB
    LOB (FILE_IMG) STORE AS SECUREFILE PL2018 (
      TABLESPACE TS_BLOB_D2018)
);

CREATE UNIQUE INDEX PK_TBL1 ON TBL1
(ARCHIVE_NAME)
TABLESPACE TS_IX;

ALTER TABLE TBL1 ADD (
  CONSTRAINT PK_TBL1
  PRIMARY KEY
  (ARCHIVE_NAME)
  USING INDEX PK_TBL1
  ENABLE VALIDATE);

POSTDATA: In the historical database would it also be necessary to partition the table?
Connor McDonald
August 29, 2018 - 7:03 am UTC

Transporting is done at the tablespace level, and the content in that tablespace must be self-contained.

So you have something like:

table data:
- data for 2018: tablespace TS_2018
- data for 2017: tablespace TS_2017
- data for 2016: tablespace TS_2016

index data:
- data for 2018: tablespace TX_2018
- data for 2017: tablespace TX_2017
- data for 2016: tablespace TX_2016

blob data:
- data for 2018: tablespace TB_2018
- data for 2017: tablespace TB_2017
- data for 2016: tablespace TB_2016

You could perform an 'exchange partition' to detach the '2016' table partition into its own table.

At that point, the tablespaces (TS_2016,TX_2016,TB_2016) form a logically complete set which could be transported to a historical database.


Victor, October 02, 2018 - 9:02 pm UTC

Gracias Tom,

Algo adicional,
En los tablespaces TS_2016 (datos) y TB_2016 (segmentos lob) contiene adicionalmente además de la información de la tabla TBL1 la información de otras tablas del mismo esquema respectivamente.

En ese escenario en el momento de copiar los datafiles tambien estaria copiando los datos de otras tablas alargando por ende el tiempo de copia. Que me recomendarias en este caso para copiar los datos historicos de una partición solo de la tabla TBL1 a otra base de datos remota.
Connor McDonald
October 03, 2018 - 3:51 am UTC

I'm glad you posted another followup in english :-)

Victor, October 02, 2018 - 9:04 pm UTC

Thanks tom,

Something additional,
In tablespace TS_2016 (data) and TB_2016 (lob segments) it additionally contains in addition to the information in table TBL1 the information of other tables of the same scheme respectively.

In that scenario at the time of copying datafiles would also be copying the data from other tables thus lengthening the time of copying. What would you recommend in this case to copy the historical data of a partition only from table TBL1 to another remote database.
Connor McDonald
October 03, 2018 - 3:54 am UTC

My point was - for this process, you could move the tables that you want to copy into a logically complete set of tablespaces. These tablespaces would contain *only* the data you want to copy.

If that is not feasible, then yes, you would be copying additional data *and* getting additional objects in your target database when you transport the data across.

More to Explore

VLDB

If you are new to partitioning, check out Connor McDonald's introduction series here.

VLDB

Documentation set on VLDB and Partitioning.