Skip to Main Content
  • Questions
  • Partition Maintenance Induced Concurrency

Breadcrumb

Question and Answer

Connor McDonald

Thanks for the question, Andy.

Asked: June 06, 2017 - 8:15 pm UTC

Last updated: June 08, 2017 - 1:36 am UTC

Version: 12.1.0.2

Viewed 1000+ times

You Asked

Hi TOMs,

A little back story:

Our application incurs many "cursor: pin s wait on x" and "library cache lock" waits during our nightly partition maintenance job. It has been expected behavior and one that we've been living happily with for quite some time. Enter the 12.1.0.2 upgrade. Now, all these sessions that were waiting for their turn to re-validate their cursors are now actually affecting the partition maintenance job and slowing it down to an unhappy pace. We've opened some SRs, patched some things, disabled some things (i.e. optimizer_adaptive_features & heatmap) and Support has finally said that we are to live with it and to change the application logic to remove this contention by forcing the partition maintenance sessions and the application sessions to play nice and wait their turns. Meaning, pause the application processing for a bit, do some DDL, turn on processing for a bit then pause, do some DDL, repeat until done.

Question:

Coding this "pause" will be a little difficult in the application since it is HUGE and there are so many pieces to pause, so in my head I thought, well, let's force a pause by LOCKing the table we will be doing partition ADD/DROPs to. For example,

session 1:
  
SQL> create table mylist (id number, n number)
  partition by list (n)
  (
    partition one values (1),
    partition two values (2)
  );    

Table created.

SQL> lock table mylist in exclusive mode;

Table(s) Locked.

Session 2:

SQL> insert into mylist values (1,2); <<< hangs while waiting for the TM lock to go away

Session 1: 

SQL> alter table mylist drop partition two;
alter table mylist drop partition two
            *
ERROR at line 1:
ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired <<< BLARG!

Session 2: 

1 row created. <<< wins the battle!



Maybe I used to know that this was expected, but as of now I'm sort of at a loss as how best to approach this. Is there a way that I can lock a table forcing all sessions to wait until I am done doing some (and actually quite a bit) DDL?

As always, thanks for your assistance and expertise!

and Chris said...

Oh boy. Looks like there's a race condition.

DDL issues a commit before and after it executes. And committing releases locks. So it looks like there's a case where the blocked insert can sneak in because the drop partition releases the exclusive lock on the table. I can reproduce this on 12.1.0.2.

This may be related to bug 21752827, MOS note 21752827.8. But in any case push back on support to sort this out!

In the meantime you can use dbms_lock to ensure either the DDL or DML runs at a given time. The following procedures will acquire and release locks:

create or replace procedure get_lock is
  v_lockhandle varchar2(200);
  v_result     number;
begin
  dbms_lock.allocate_unique('DDL_lock', v_lockhandle);

  v_result := dbms_lock.request(v_lockhandle, dbms_lock.x_mode);

  if v_result <> 0 then
    raise_application_error(-20001, 'Unable to get lock');
  end if; 
end get_lock;
/

create or replace procedure release_lock is
  v_lockhandle varchar2(200);
  v_result     number;
begin

  dbms_lock.allocate_unique('DDL_lock', v_lockhandle);

  v_result := dbms_lock.release(v_lockhandle);

  if v_result <> 0 then
    raise_application_error(-20001, 'Unable to release lock');
  end if; 

end release_lock;
/


These persist across commit by default. So you call get_lock before doing your DDL and release_lock after it completes:

exec get_lock;

alter table mylist drop partition two;

exec release_lock;


And similarly for your DML:

exec get_lock;

insert into mylist values (1,2);

exec release_lock;


Only one of these sessions will be able to proceed. This will of course mean that concurrent inserts will block each other too. Which may cause you some big concurrency issues...

So you'll probably want to add some checks so you only call the get/release lock procedures for your DML during your DDL maintenance window.

Rating

  (1 rating)

Is this answer out of date? If it is, please let us know via a Comment

Comments

DBMS_LOCK.X_MODE!

Andy Klock, June 07, 2017 - 3:04 pm UTC

Thanks Chris. I'd not even considered dbms_lock. I tested your idea, but I didn't make it much further, but it's possible that I'm not doing it right. For example,

Session 1:
SQL> exec get_lock;

PL/SQL procedure successfully completed.


Session 2 (slips in a fast one...):
SQL> insert into mylist values (1,2);

1 row created.


Session 1:
SQL> alter table mylist drop partition two;
alter table mylist drop partition two
            *
ERROR at line 1:
ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired

Need some ddl_lock_timeout...

Now, I do get a little farther (which is probably where you were leading me to) if I call the LOCK TABLE immediately after dbms_lock.x_mode:

Session 1:
SQL> exec get_lock;

PL/SQL procedure successfully completed.

SQL> lock table mylist in exclusive mode;

Table(s) Locked.



After which Session 2 waits until Session 1 is done.

Session 2:
SQL> insert into mylist values (1,2);
insert into mylist values (1,2)
            *
ERROR at line 1:
ORA-14400: inserted partition key does not map to any partition



However, after reflecting a little more on our environment the majority of our cursor pins and library cache locks are really on SELECT statements. So all of this is probably for naught...

I'll be sure to tell Support you told them to get back to work. Thanks again for your help. Love TOMs.
Connor McDonald
June 08, 2017 - 1:36 am UTC

I think Chris's point was that if you adopt a serialisation mechanism, then *all parties* who want to compete for the resource need to embrace it.

In those sites that have a plsql wrapper around their code, this is of course trivial - you just add dbms_lock where appropriate.

More to Explore

VLDB

If you are new to partitioning, check out Connor McDonald's introduction series here.

VLDB

Documentation set on VLDB and Partitioning.