Skip to Main Content
  • Questions
  • Advanced Queuing to pass data from one database to another?

Breadcrumb

Question and Answer

Tom Kyte

Thanks for the question, Robert.

Asked: February 16, 2005 - 7:35 pm UTC

Last updated: February 18, 2005 - 4:07 pm UTC

Version: 8.1.7 or 9.2.0.5

Viewed 1000+ times

You Asked

Hi Tom,

Consider an application which sends/receives data from one database to another...(e.g. orders are inserted into a table in DB1, then an application on DB2 reads those orders and processes them.. and other communication beteen the two databases).
Would this be a good candidate for AQ?
My understanding about AQ is that it can be used to 'buffer' data between databases, so that if one database/connection is down the application won't crash, the necessary data will simply be enqueued into the queue and when the database/connection is back up, the other database can dequeue this data at its leisure.

Is my understanding of this correct?

Thanks,

Robert.


and Tom said...

sounds very much like "yes" to me.

AQ (queueing in general) is a method to provide (in general)

o prioritized
o ordered
o message payloads
o at least once
o at most once (if desired)


we use it for replication (advanced replication and streams replication)



Rating

  (5 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

A little more analysis?

Doug, February 17, 2005 - 10:51 pm UTC

So I was wondering if you could go into more depth as to why I would want to use this option. If I have a dblink and a site goes down, generally I can restart where I left off after the db comes up if it has crashed or whatever. Is this similar logic to MQ? I want things to restart auomatically?

Tom Kyte
February 18, 2005 - 7:33 am UTC

if you have an end user (sure hope you do), does the end user want

a) to have to wait for all databases to be up to place an order
b) to just have their database up to place an order?

does the end user want to wait for

a) a local transaction and remote transaction to take place, along with the mandatory two phase commit or...
b) a local transaction with a simple commit

can the database you are replicating into or consolidating into handle

a) its load PLUS all of the load from the other machine or...
b) its load PLUS a "manageable, controllable flow of data from the other machine"


It would be more fair to say that MQ is like AQ yes. MQ is a stripped down db2 database used as a stable store (it is a database, needs to be backed up, recovered and so on -- but you cannot use it as a database, so it is yet another cog in the machine to manage). AQ provides a message queue inside the database -- the database you are problably already transacting against in the first place.


It is all about providing to the end user a more available, apparently faster, more reliable place to do their work. Slow things happen in the background, failures of systems are tolerated -- expected even, but gracefully dealt with.

Can AQ address data conflict issues?

Sami, February 17, 2005 - 10:52 pm UTC

Dear Tom,
Data conflict is the major issue in Advanced Replication (in master-master environment). This conflict occurs becoz of the time delay to push the transactions from one DB to others.
Is there a way to avoid this conflict using AQ ?

Say for example we have 100 tables in replications (Master-master between NJ and NY). The application can tolerate 2-5 mins replication delay in 95% of the tables but 5 tables should be replicated in near real-time.

Can I use AQ and propogate the changes in less than a Min?
I would like to know your view on this.

Tom Kyte
February 18, 2005 - 7:36 am UTC

this conflict occurrs BECAUSE two transactions can take place against the same logical data at precisely the same point in time (time delay is not the factor in as much as "we are asyncronous").

AQ is a foundation technology.
AQ cannot be used to "avoid this"

Only SYNCRONOUS replication (we update both locations in the same transaction) or rules that prevent the same bits of data being updated in more than one location at a time are able to "prevent" this.

AQ can propagate as fast as you like, time has no bearing on this at all. In a split instant you have update conflicts -- I'll update "where x=5" on site 1 -- you update "where x=5" on site 2 -- at the same instant. No time delays involved.




Where is a good example of this?

Robert, February 18, 2005 - 12:33 am UTC

Tom,

Can you point me to a simple example that uses AQ to loosly couple databases... and that enqueue/dequeue 'real' data (e.g. for updating Orders tables with actual orders).

Many of the Oracle examples appear to be using AQ in some kind of esoteric or lightweight manner or else are incorporating it into front-end (e.g.web) applications.

How about a simple, basic, robust database-to-database plain old data example?

Thanks,

Robert.

Tom Kyte
February 18, 2005 - 7:55 am UTC

replication -- we use it there.

you see it here on asktom. true, I use dbms_jobs as a poor mans AQ but it is precisely the same concept. When I do stuff, it generates a message (a job). That message -- "document=4321432143 was modified, notify interested parties". The database I'm replicating to? SMTP -- I send an email, after I commit.


robust database to database plain old data example -- replication. done. lots of examples/documentation on that. AQ is the building block (not the end solution, an enabling technology).



Not the same thing.....

Robert, February 18, 2005 - 10:11 am UTC

Tom,

I am not understanding how what you are saying is answering my specific question.... :)

I hope this diagram is explaining my concept....

My application is not necessarily 'replicating' table-to-table, 1-to-1...

but....
(1) all updates occurring on Database 1 must be acted on by Database 2 (or at least considered).
(2) A broken database communication must not crash the application.


Database 1 Database 2
(may be up or down) \\ (may be up or down)
\\
App1 \\
On Update AQ \\ update
Trigger? QUEUE(?) \\ App2 table(s)
+-----+ enqueue +------+ \\ +-----+ dequeue +-----+
| |-------> | | \\ | | -------> | |
| | | | \\ | | | |
+-----+ +------+ \\ +-----+ +-----+

Am I understanding?
Is this a legitimate use of AQ?
(I don't want to build a mind boggling Rube Goldberg contraption with 1000 moving parts when 1 or 2 moving parts will do. :)

Thanks,

Robert.

Tom Kyte
February 18, 2005 - 1:58 pm UTC

that just described replication to a tee

(1) and (2), we call that "replication"


but if you just want to tell database 2 'hey, database 2, this is database 1, we just did "X", if you care -- do something about that fact'

then your picture applies just as well.


Think of AQ just like you think of EMAIL (that is what it is with lots of fancy stuff).

If you were to send email to me, what happens?

a) you perform a local transaction of compose email and send to your smtp server (the database) and commit. If my system is up or down -- you don't care. If your network to the outside world is up or down -- you don't care. The only thing that needs to be up is your database (smtp server)

b) after you commit (give the email to your smtp server), it will try to relay it to me. If the network is down, it'll keep trying. If the network send fails partway through the send, it'll retry it later. Eventually the message gets to my system.

c) I read it and act on it (doing whatever..)


That describes AQ as well. Just "store and forward", just messages. You can build whatever you like on top of it.


But your picture looks suspiciously like "replication" to me -- eg: STREAMS -- it does all of the above and if you want to do the update yourself, streams could call you (eg: you need only write the "apply" routine on database 2)

Time to digest information

Robert, February 18, 2005 - 4:07 pm UTC

Tom,

Thanks to your back and forth clarification I think I am becoming more clear on the issues involved and the technology to achieve my solution.
Especially the fact that AQ is a 'low-level' technology on which other things are built.
I will start investigating replication (and streams on 9i).

Thanks again,

Robert.