Skip to Main Content

Breadcrumb

Question and Answer

Chris Saxon

Thanks for the question, Ravi.

Asked: July 20, 2016 - 8:30 pm UTC

Last updated: July 25, 2016 - 8:49 am UTC

Version: 11g/12c

Viewed 1000+ times

You Asked

Hello AskTom Team,

Hope you could help me with this.

Business case:

1) We have system of records which we provide to customers. We refresh this data on demand (could be push or pull) on customer's on-premise instance
2) Cusotmer can override (update) any of attributes we provide OR they can create their own system of records (insert)
3) Cusotmer should be able to revert back to original value of system of record that we provide (step 1)

Our legacy system is implemented this way:

Note: This is a very simplified representation of how it is actually implemented. Actual implementation is much complicated than this but this would demonstrate the case

-- over all idea
1) Create a table that holds system of records
2) Create replica of the table which holds modified records or created records
3) create a view on top of these two tables to show combination of original/modified/customer created data

Folllwing demonstrates the case:

a) Table DEPT_PUBLIC has system of records (original golden set)
b) Table DEPT_PRIVATE has modified system of records AND newly created records by customer
c) positive DPETNO are original records (generated by oracle SEQUENCE)
d) negitive DEPTNO are created by customer (generated by oracle SEQUENCE)
e) all the modified and newly created records are inserted in DEPT_PRIVATE
- we do not actually UPDATE data in DEPT_PUBLIC as we want to keep original golden data as-is
- first time UPDATE of any record is actually INSERT into DEPT_PRIVATE
- only changed attributes are reflected in DEPT_PRIVATE, un-changed attrubutes have NULL value
- subsequent updates would be UPDATES in DEPT_PRIVATE table
- INSERT of new recors would be INSERT in DEPT_PRIVATE table with new -ve DEPTNO
- subsequent updates of private records are in DEPT_PRIVATE table

DROP TABLE DEPT_PUBLIC;

-- create system of records table
CREATE TABLE DEPT_PUBLIC
       (DEPTNO NUMBER,
        DNAME VARCHAR2(100),
        LOC VARCHAR2(100) );
        
-- create primary key
ALTER TABLE DEPT_PUBLIC  ADD CONSTRAINT DEPT_PUBLIC_PK PRIMARY KEY ( DEPTNO ) ;
        
--insert system of records data
INSERT INTO DEPT_PUBLIC VALUES (10, 'ACCOUNTING', 'NEW YORK');
INSERT INTO DEPT_PUBLIC VALUES (20, 'RESEARCH',   'DALLAS');
INSERT INTO DEPT_PUBLIC VALUES (30, 'SALES',      'CHICAGO');
INSERT INTO DEPT_PUBLIC VALUES (40, 'OPERATIONS', 'BOSTON');

COMMIT;

SELECT * FROM DEPT_PUBLIC;

DROP TABLE DEPT_PRIVATE;

-- create replica of DEPT_PUBLIC
CREATE TABLE DEPT_PRIVATE
        (DEPTNO NUMBER,
        DNAME VARCHAR2(100),
        LOC VARCHAR2(100) );
        
-- create Primary Key
ALTER TABLE DEPT_PRIVATE  ADD CONSTRAINT DEPT_PRIVATE_PK PRIMARY KEY ( DEPTNO ) ;
        
-- modify/update ACCOUNTING to MY_ACCOUNTING
-- only changed attributes are reflected in DEPT_PRIVATE table, un changed attrubutes are left as NULL
INSERT INTO DEPT_PRIVATE VALUES (10, 'MY_ACCOUNTING', NULL);

-- modify/update OPERATIONS to MY_OPERATIONS
-- only changed attributes are reflected in DEPT_PRIVATE table, un changed attrubutes are left as NULL
INSERT INTO DEPT_PRIVATE VALUES (40, 'MY_OPERATIONS', NULL);

--insert a new custom record MY_NEW_DEPARTMENT
INSERT INTO DEPT_PRIVATE VALUES (-100, 'MY_NEW_DEPARTMENT', 'MY_LOCATION');

COMMIT;

SELECT * FROM DEPT_PRIVATE;

CREATE OR REPLACE VIEW DEPT (DEPTNO,DNAME,LOC) AS
SELECT pub.DEPTNO,pub.DNAME,pub.LOC
  FROM DEPT_PUBLIC pub
 WHERE NOT EXISTS
    (SELECT 1
    FROM DEPT_PRIVATE pri
    WHERE pub.DEPTNO=pri.DEPTNO
    )
UNION ALL
SELECT DEPTNO,DNAME,LOC
  FROM 
   (SELECT pri.DEPTNO,
           COALESCE(pri.DNAME,pub.DNAME) DNAME,
           COALESCE(pri.LOC,pub.LOC) LOC
    FROM DEPT_PRIVATE pri LEFT JOIN DEPT_PUBLIC pub ON pub.DEPTNO = pri.DEPTNO
    WHERE SIGN(pri.DEPTNO )!= -1
   )
UNION ALL
SELECT pri.DEPTNO,pri.DNAME,pri.LOC
  FROM DEPT_PRIVATE pri
  WHERE SIGN(pri.DEPTNO )= -1;
  
select * From DEPT ORDER BY DEPTNO;


So far so good. The problem arises when customer wants to NULLyfy a value.
The intention is, instead of modifying a value the customer want to make the value as NULL.
In this case COALESCE would not work.

Question:

1) How to handle NULL update of original record
2) if you have to re-design this, how would you do it?

Please let me know if you need any further clarifications.

and Chris said...

Well why don't you insert all the values into dept_private when someone "updates" the public copy then?

Then you can read the values out of private instead of faffing around comparing with the public table.

Your view is then:

SELECT pub.DEPTNO,pub.DNAME,pub.LOC
  FROM DEPT_PUBLIC pub
 WHERE NOT EXISTS
    (SELECT 1
    FROM DEPT_PRIVATE pri
    WHERE pub.DEPTNO=pri.DEPTNO
    )
UNION ALL
SELECT pri.DEPTNO,pri.DNAME,pri.LOC
  FROM DEPT_PRIVATE pri;


But the whole process sounds bizarre to me. If customers can update the original data, why not just let them? Have a single dept table for them to do as they want.

You have the original values in your database. If they ever need these resetting, you can load your data into a temporary table and do a merge.

Or to the more general question:

If a customer updates a public value in their database, why do they ever need it resetting?

If this is some kind of controlled data they're not allowed to update, you could look into something like VPD to prevent this.

Rating

  (6 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

Ravi B, July 21, 2016 - 4:15 pm UTC

Well why don't you insert all the values into dept_private when someone "updates" the public copy then?

A: Because, we have data updates each day which is pulled into customer database. The data update could be several hundred tables and several GB of data.

Say for example, customer wants to update one column, say "description" but leave rest of the columns as-is. Rest of the columns potentially could be changed by us on a daily basis. Customer wants to get latest data from us for rest of the columns. It is very difficult to keep track of potentially several thousand columns which could be selectively updated, and refresh only rest of the columns on a daily basis. Instead is easy for us to do a complete refresh of public data. If all the data is in one table, at some point we do not know the change is due to customer or the change is genuinely coming from our side.

You have the original values in your database. If they ever need these resetting, you can load your data into a temporary table and do a merge

A: Same reason as above; difficult to track which columns has been modified and selectively do merge on rest of the columns.

If a customer updates a public value in their database, why do they ever need it resetting?

A: There are some use cases where user might want to revert a particular column to original system of record. We also show on the UI, the value from which it has been updated.
Chris Saxon
July 22, 2016 - 7:34 am UTC

I'm not getting it. Why are you sending updates to the customer's database every day?

It sounds to me like you need to rethink the strategy. But I don't understand enough about the process to comment exactly how.

Ravi B, July 22, 2016 - 9:33 am UTC

I'm not getting it. Why are you sending updates to the customer's database every day?

Because it is market research data. Data is modified, added and removed several times per day. Data is difficult to collect from 100's of different sources, research validity, cleanse and curate it. For example, commodity pricing data.
Chris Saxon
July 22, 2016 - 10:25 am UTC

But if your data is the master, the "golden copy", why do you allow customers to overwrite it then?

Ravi B, July 22, 2016 - 3:14 pm UTC

Let me give one hypothetical use case as i cannot discuss actual ones:

Let us say it is pricing data, and we say the price of a particular item is $1000. But the customer payed $400 because he got a discount due to a promotion. In his financial reporting, he doesn't want the pricing to be inflated by $600. He would go a update the price to $400. He runs out of promotion period and he starts paying original price. He goes and "reverts" his updates.

For example we categorize a particular item to be in "Finance" but customer thinks the item should be part of "sales & finance" he goes and updates the category to predefined category "sales & finance". If we do not provide a predefined category customer doesn't find, he would go and create one and assign this category.
Chris Saxon
July 25, 2016 - 8:49 am UTC

This is clearly a complex business problem. This forum isn't the appropriate place for us to give meaningful advice about how you best resolve this. There's too much we don't know about your requirements and constraints.

Chuck Jolley, July 22, 2016 - 5:40 pm UTC

If you are constantly refreshing and sometimes undoing customer changes then how do you know when NOT to over-write a modification the customer has made?

Ravi B, July 23, 2016 - 6:56 am UTC

That is exactly why we have two separate tables and COALESCE. If there is a modification COALESCE would find out which row or column is changed. Customer would never overwrite the data we provided. They think they are modifying the data but actually they are not, we mask it using a view. They can revert back to standard values provided by us when ever they want.

Ravi B, July 25, 2016 - 5:19 pm UTC

Totally agree with Chris. This is a very complex problem and took a while to get to a point where it is now, with bugs like this. We are moving out of oracle soon to noSQL due to very nature of the business model and also high licensing costs. We have proposed solution with noSQL hopefully it works :)

More to Explore

Security

All of the vital components for a secure database are covered in the Security guide.