Skip to Main Content
  • Questions
  • parallel statement queueing - how to synchronize sessions?

Breadcrumb

Question and Answer

Connor McDonald

Thanks for the question, Bernd.

Asked: July 14, 2016 - 8:29 am UTC

Last updated: July 22, 2016 - 10:24 am UTC

Version: 12.1.0.2

Viewed 1000+ times

You Asked

Hi Tom-Team,

I want to process statements in parallel by setting "parallel_degree_policy=auto". They are a mixture of DML and DDL statements. In this context, the order in which statements execute, is undefined. I know that I can amend that (partially) with DBMS_RESOURCE_MANAGER.BEGIN_SQL_BLOCK/END_SQL_BLOCK, but that is not sufficient for me.

What I would need is a "wait" command - halt script execution until all parallel sessions are finished, and then continue. Doing this in one or two locations of my script would solve my problem.

Does such a wait-command exist? Or alternatively, can SQL_BLOCKs be nested?

-- Update starts here.

I have added my script at https://livesql.oracle.com/apex/livesql/s/dkwk2cauf1velr5jei3xk55wh

Probably it's better I explain what I want to do in symbolic Terms (The General Setting is: a new database is loaded from older databases with dump files. The imported dumpfiles do not have Basic compression. To achieve Basic compression, the data are then direct-path-copied with an intelligent order by clause to their final Location. I start only one session; it is left for the database to start parallel slaves as required.) I set

alter session set PARALLEL_DEGREE_POLICY=AUTO;
alter session enable parallel dml;
alter session enable parallel ddl;

to enable parallel processing. Then (the Iteration over partitions is necessary because otherwise Transactions would become too large. During this operations, target Indexes are set unusable):

-- Loop 1: copy source to target.
for my source-partitions Loop
DBMS_RESOURCE_MANAGER.BEGIN_SQL_BLOCK;
copy source Partition to target with intelligent order by for Maximum compression;
erase source Partition;
DBMS_RESOURCE_MANAGER.END_SQL_BLOCK;
end Loop;

-- Loop 2: Rebuild target Indexes
for my target_index_partitions Loop
rebuild index Partition;
end Loop;

Because of PARALLEL_DEGREE_POLICY=AUTO the Statements are expected to be queued and executed in parallel slave sessions as they become available (therefore their order of processing of Statements may be different from the order of submission). The BEGIN/END_SQL_BLOCK should ensure that source data are not erased before they are copied to the target.
After that, index partions are rebuild in Loop 2. I have noticed that in particular in context of index (Partition) rebuilds the database uses Statement queueing and execution in parallel slaves rather heavily, and that is quite welcome. The only Thing that SHOULD NOT HAPPEN is that a "rebuild" Statement from Loop 2 is executed before the corresponding Partition is loaded in Loop 1. So what I would Need is, between Loops 1 and 2 to wait until all parallel slaves from Loop 1 are finished.

and Chris said...

If parallel statements are queued, then Oracle Database processes them in a strict first-in, first-out scheme. So if your statements are queued, whichever started first will be processed first.

You can read more about this at:

https://blogs.oracle.com/datawarehousing/entry/auto_dop_and_parallel_statemen

But your whole process seems bizarre to me! You don't get any compression by "inserting with an intelligent order by". You get compression by enabling compression. Your script doesn't do that.

https://oracle-base.com/articles/11g/table-compression-enhancements-11gr1

In any case, any advantage you get with the "intelligent ordering" will be lost over time. If you're loading into heap tables then Oracle will add new rows wherever there's space.

Rating

  (5 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

Problem not addressed

Bernd Guenther, July 21, 2016 - 7:39 am UTC

(1) Of cource the tables have compression enabled. Basic table compression utilizes repeated values, so in order to achieve Maximum compression you must ensure that you have many repeats. We do that by ordering data along fields of low cardinality, hence we get many repetitions. (We are talking of an Initial load in a database Migration).

(2) About parallel, the Problem is the following: Suppose you process three Statements A, B, C and submit them in this order. As you say, they will be queued in this order. Now suppose each of them will have degree of parallelism 2 and you have six parallel Servers available, so all three will start processing at once. Now what do you do if you Need to finish Statement A before Statement C should start?
Connor McDonald
July 21, 2016 - 8:12 am UTC

If you submit three statements in the same session, A still has to finish before B starts! Even if A is queued, B only starts once A completes.

Statement queuing only matters across sessions.

I cannot confirm that

Bernd Guenther, July 21, 2016 - 8:53 am UTC

Hi Chris,
I cannot confirm that! You might consult e.g. metalink doc 1265930.1. Also I know from Oracle 11.2 If I submit, say, index Partition rebuild Statements with parallel degree 1, but have parallel ddl and parallel degree policy Auto, then several Statements will run simultaeously in parallel sessions.
Chris Saxon
July 21, 2016 - 3:46 pm UTC

A single session can only run one statement at a time. The first statement must complete before Oracle tries to start the second.

Think of a session with two updates:

update t1 set x = 1;
update t2 set x = 2;


If the first is blocked due to an uncommitted transaction, Oracle doesn't start the second! It waits. Parallel statement queuing is a similar concept.

If you think otherwise please post a test case *showing* that Oracle executes the statements "out of order".

TRANSFORM option with Datapump

Rajeshwaran, Jeyabal, July 21, 2016 - 2:36 pm UTC

(The General Setting is: a new database is loaded from older databases with dump files. The imported dumpfiles do not have Basic compression. To achieve Basic compression, the data are then direct-path-copied with an intelligent order by clause to their final Location. I start only one session; it is left for the database to start parallel slaves as required.

If the goal is to have "Basic Compression" during import then it is possible to achieve that using "TRANSFORM" option with Datapump imports, also making use of PARALLEL parameter would do parallel dmls too.

Am i missing something here?

http://docs.oracle.com/database/121/SUTIL/GUID-64FB67BD-EB67-4F50-A4D2-5D34518E6BDB.htm
Chris Saxon
July 21, 2016 - 3:48 pm UTC

I think the point is they want to force a particular order for "optimal" compression. Data pump doesn't do this.

Attribute Clustering

Rajeshwaran, Jeyabal, July 22, 2016 - 4:29 am UTC

I think the point is they want to force a particular order for "optimal" compression. Data pump doesn't do this.

since they are in 12c, does attribute clustering helps them here?

http://docs.oracle.com/database/121/DWHSG/attcluster.htm#CCHEEJFD
Connor McDonald
July 22, 2016 - 7:12 am UTC

One of the benefits of attribute clustering is:

Compression can be improved because, with clustering, there is a high probability that clustered columns with the same values are close to each other on disk, hence the database can more easily compress them.

So, possibly

Thanks for pointing out attribute clustering

Bernd Guenther, July 22, 2016 - 7:51 am UTC

Hi Rajeshwaran and Chris,

thanks for pointing out Attribute clustering. I was not Aware of that but it sounds interesting.

About the parallel stuff; this particular strategy seems to be new in Oracle 11.2. What you do is e.g. enable parallel ddl, parallel degree policy Auto, enough parallel Servers and submit two index rebuild Statements. You will notice that they execute simultaneously.
See Home / Database / Oracle Database Online Documentation 11g Release 2 (11.2) / Data Warehousing and Business Intelligence / Database VLDB and Partitioning Guide/ 8 Using Parallel Execution / Parallel Statement Queuing.

DocuOra11g/server.112/e25523/parallel002.htm_hash_BEIBACGC

I will try to provide a testcase.
Connor McDonald
July 22, 2016 - 10:24 am UTC

Where precisely in that document does it say Oracle will start processing a second statement in the same session while the first is in the parallel queue?

I can't see anything that says that!

More to Explore

Data Pump

All of the database utilities including Data Pump are explained in the Utilities guide.