Skip to Main Content
  • Questions
  • Synchronizing database sequences during manual data replication

Breadcrumb

Question and Answer

Connor McDonald

Thanks for the question, Evelina.

Asked: June 03, 2019 - 10:07 pm UTC

Last updated: June 12, 2019 - 1:12 am UTC

Version: 12c

Viewed 1000+ times

You Asked

Hi Tom, Experts,

I am in a need to replicate manually all objects from one schema to another (manually, because my schemas, both source and target, reside in the cloud and I have no access to any OS level utility, nor sql*net access to the database to connect with a client tool).
The goal is that I create a regular backup of my data that I am refreshing say weekly, and in the case of a disaster (eg even user error) I can easily switch the database schema under my application from the source to the target schema.

I'd like to ask what is an elegant solution for keeping my sequences in sync between the source and the target. In practice I'd like to use highly automated solution that can be dbms_scheduler-scheduled and does not require any manual intervention and can be executed in both full/initial and incremental way.

I can grant select from the source to the target schema, and use insert as select - so far so good.

My problem is keeping the sequences that populate the primary keys in sync between the source and the target, so that
1. In the target the primary keys already populated in the tables remain the same as in the source, to maintain the FK integrity
2. When I try to insert new record in the target, the corresponding sequence provides a number that comes after the last assigned number in the source

I am finding it even a little trickier with the tables using the 12c Identity column feature, as the system generated sequences don't even have the same names in the source and the target schema (if at all I wanted to maintain them manually!)

Can you suggest an elegant way to keep my sequences in sync during such manual "replication"?
I have couple of ideas (sharing further down), but not finding any of them particularly elegant.

Here is a very simple test case (with Identity column):

-- creating the table in the source schema and populating couple of records
CREATE TABLE  "AI1" 
   ( "AI1ID" NUMBER GENERATED BY DEFAULT AS IDENTITY MINVALUE 1 MAXVALUE 9999999999999999999999999999 INCREMENT BY 1 START WITH 1 CACHE 2 NOORDER  NOCYCLE  NOKEEP  NOT NULL ENABLE, 
 "NAME" VARCHAR2(10), 
  CONSTRAINT "AI1_PK" PRIMARY KEY ("AI1ID")
  USING INDEX  ENABLE
   )
/


insert into ai1 (name) values ('test1'); 
/
insert into ai1 (name) values ('test2'); 
/
insert into ai1 (name) values ('test3'); 
/



-- now recreating the object in the other schema (for this test it doesn't really matter if it is the same or different schema, as the identity column system-generated sequence will have a unique name anyway) and copying over the data with insert as select:

CREATE TABLE  "AI2" 
   ( "AI2ID" NUMBER GENERATED BY DEFAULT AS IDENTITY MINVALUE 1 MAXVALUE 9999999999999999999999999999 INCREMENT BY 1 START WITH 1 CACHE 2 NOORDER  NOCYCLE  NOKEEP  NOT NULL ENABLE, 
 "NAME" VARCHAR2(5), 
  CONSTRAINT "AI2_PK" PRIMARY KEY ("AI2ID")
  USING INDEX  ENABLE
   )
/

insert into ai2 select * from ai1;
/


Now, suppose my source schema got broken and I want to switch to the target schema.

If I try to insert a row in AI2, it will error out as the data default sequence on the primary key column has not got advanced in sync with the previously inserted data:


insert into ai2 (name) values ('test4'); 


ORA-00001: unique constraint (EKTEST.AI2_PK) violated
ORA-06512: at "SYS.DBMS_SQL", line 1721

How can avoid this?

What I have been thinking of in case of identity columns:

1. Adjust my table creation ddl to specify the "start with" as a value greater than the largest value of the primary key of the source table - this will not help in case of incremental insert

2. Add a cycle to lift up the sequence to the same max value - well, not really nice

3. Add a cycle to insert dummy records in the target table until the sequence reaches the max value in the source table, then truncate the table and insert-as-select the real records - even less nice and again not good in case of incremental insert

What is your expert recommendation?

many thanks in advance!


and Connor said...

This is a much much larger task then you think. There are locking issues, data ordering to ensure referential integrity, and many many more things that complicate things.

You'd need to track grants, synonyms, views, plsql, all other DDL...

You want to look at something like Goldengate for this.

If you really have to do it manually, and can't use DataPump, then I'd recommend simply doing create-table-as-select because tracking incremental changes is a *huge* undertaking.

Rating

  (3 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

Thank you for the input, suggested workaround does not really work ..

Evelina Karolyi, June 05, 2019 - 7:37 am UTC

First of all, thank you for the detailed response

I indeed gathered that the task is not around tables only, I am using the dbms_metadata APIs and taking care of "replicating" all the objects in my source schema and then inserting as select, carrying over all the PK values. I am turning off the PK generation triggers off during the insertion so that I can preserve the original PK values - for non-Identity type PK columns.

Create table as select did not appear to me a solution for replicating a table with Identity column.
The PK column in the copy is not an identity column. And not even a PK.
Consider my original test case above, if I did:

create table ai2 as select * from ai1


here is what the 2 tables look like in practice:

AI1:

Column Name Data Type Nullable Default Primary Key
AI1ID NUMBER No "EKTEST"."ISEQ$$_9703123".nextval 1
NAME VARCHAR2(10) Yes - -


AI2:

Column Name Data Type Nullable Default Primary Key
AI1ID NUMBER No - -
NAME VARCHAR2(10) Yes - -


This unfortunately is nowhere close to what I need. I need my copy to look exactly the same as the source table (apart from the system generated sequence name).

So if we only concentrate on tables, and say only on tables with identity columns, consider I can ignore any locking issues as my goal is to have a somewhat recent copy of my data and losing all my data is much bigger loss than losing few most recent changes, is there no elegant way to keep the target sequence in sync with the source sequence (for the moment when creating the replica) ? Even let's forget incremental changes tracking but think about recreating the table from scratch.

If there isn't, perhaps we'd like to consider once an enhancement on the GENERATED BY DEFAULT Identity column to actually advance the system generated sequence to the Identity column value of the inserted records, when value is being supplied (not only when NULL is supplied).
Just an idea

thanks and regards
Evelina

PS: And indeed, DataPump or GoldenGate are not options in my case. (unfortunately)
Connor McDonald
June 11, 2019 - 6:07 am UTC

Maybe just use a two step process - DEFAULT and then LIMIT VALUE, eg

SQL> create table t ( x int generated as identity, y int, z int );

Table created.

SQL> insert into t (y,z)
  2  select rownum, rownum
  3  from dual
  4  connect by level <= 10;

10 rows created.

SQL> commit;

Commit complete.

SQL> create table t1 ( x int generated by default on null as identity , y int, z int );

Table created.

SQL> insert into t1 select * from t;

10 rows created.

SQL> alter table t1 modify x  generated always as identity (start with limit value) ;

Table altered.

SQL> insert into t1 (y,z) values (0,0);

1 row created.

SQL> select * from t1;

         X          Y          Z
---------- ---------- ----------
         1          1          1
         2          2          2
         3          3          3
         4          4          4
         5          5          5
         6          6          6
         7          7          7
         8          8          8
         9          9          9
        10         10         10
        11          0          0  <==== starts at 11

11 rows selected.

SQL>


Maybe useful, maybe not

Tubby, June 05, 2019 - 3:50 pm UTC

Perhaps the use of sys_guid would solve the OPs issues, though I'm not sure if that's an acceptable option for them or not.

https://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:9539286100346029540

"
And remember: performance is only one consideration. GUIDs will almost certainly use more storage. But they're also globally unique. Which may be useful if you're replicating data elsewhere.
"

Cheers,

Most helpful!

Evelina Karolyi, June 11, 2019 - 6:54 am UTC

The 2-steps approach with DEFAULT and then LIMIT VALUE looks quite promising. I wasn't aware of the LIMIT VALUE option - even if I saw it earlier I wouldn't have thought it is this what it is good for ..
I will test more thoroughly - thanks so much for the hint!

As per the sys_guid() option - yes, this would have been even more useful if I knew about it from the start! I am using Identity on my less fast-growing tables, so storage wouldn't have been a concern, but I'll have to thik it twice before I decide on rebuilding exisiting tables with live data to replace Identity with sys_guid()-generated ids.

Thank you both for the helpful inputs!
kind regards
Evelina
Connor McDonald
June 12, 2019 - 1:12 am UTC

glad we could help

More to Explore

Administration

Need more information on Administration? Check out the Administrators guide for the Oracle Database