Skip to Main Content
  • Questions
  • Datapump SQL minitoring with SQL (not PL/SQL)

Breadcrumb

Question and Answer

Connor McDonald

Thanks for the question, Mette.

Asked: March 01, 2018 - 11:43 am UTC

Last updated: March 05, 2018 - 4:00 pm UTC

Version: 12.1

Viewed 1000+ times

You Asked

I'm on 11.2 and 12.1 and would like to monitor datapump sessions with pure SQL (I am not allowed to create objects on the databases).

I can monitor using:

select ss.sid
, s.job_name
, j.state
--, j.job_mode
--, j.degree
, dbms_rcvman.Sec2DisplayTime(l.time_remaining) remain
--, round((l.sofar/l.totalwork)*100,2) done
--, s.owner_name
, sq.sql_text
, l.message
, l.totalwork
, l.sofar
, round((l.sofar/l.totalwork)*100,2) done
, l.sql_id
, ss.event
, ss.state
, ss.seconds_in_wait
from dba_datapump_jobs j
left join dba_datapump_sessions s on (s.job_name = j.job_name)
left join v$session ss            on (ss.saddr = s.saddr)
left join v$sql sq                on (ss.sql_id = sq.sql_id)
left join v$session_longops l     on (l.sql_id = ss.sql_id)
WHERE 1=1
and ss.module='Data Pump Worker'
--and l.time_remaining > 0
order by sid, l.start_time
;

-----------

But I would like to have information from the individual workers - as in ie .... the status command from attaching to the datadump session. I can get it using PL/SQL like this, but I'm looking for pure SQL without creating objects:

DECLARE
  ind NUMBER;              
  h1 NUMBER;               
  percent_done NUMBER;     
  job_state VARCHAR2(30);  
  js ku$_JobStatus;        
  ws ku$_WorkerStatusList; 
  sts ku$_Status;          
BEGIN
h1 := DBMS_DATAPUMP.attach('SYS_IMPORT_FULL_01', 'SYSTEM');
dbms_datapump.get_status(h1,
           dbms_datapump.ku$_status_job_error +
           dbms_datapump.ku$_status_job_status +
           dbms_datapump.ku$_status_wip, 0, job_state, sts);
js := sts.job_status;
ws := js.worker_status_list;
      dbms_output.put_line('*** Job percent done = ' || to_char(js.percent_done));
      dbms_output.put_line('restarts - '||js.restart_count);
ind := ws.first;
  while ind is not null loop
      dbms_output.put_line( 
           ws(ind).worker_number || ',' 
        || ws(ind).state || ',' 
        || ws(ind).schema || ',' 
        || ws(ind).NAME || ',' 
        || ws(ind).object_type || ',' 
        || ws(ind).degree || ',' 
        || ws(ind).percent_done || ',' 
        || ws(ind).COMPLETED_ROWS
        );

 --   dbms_output.put_line('rows completed - '|| ws(ind));
    ind := ws.next(ind);
  end loop;
DBMS_DATAPUMP.detach(h1);
end;
/


Can I in 12.1 make an inline function which does the api calls ? Or can I use an internal view for getting the status of the workers ?

regards Mette from DK

and Chris said...

Your example doesn't create any objects? It's just an anonymous block.

Anyway, starting in 12.1 you can define PL/SQL functions in the with clause. So you could do something like:

with 
FUNCTION get_job ( 
  job_name varchar2, owner_name varchar2 
) RETURN INT AS
  ind NUMBER;              
  h1 NUMBER;               
  percent_done NUMBER;     
  job_state VARCHAR2(30);  
  js ku$_JobStatus;        
  ws ku$_WorkerStatusList; 
  sts ku$_Status;          
BEGIN
h1 := DBMS_DATAPUMP.attach(job_name, owner_name);
dbms_datapump.get_status(h1,
           dbms_datapump.ku$_status_job_error +
           dbms_datapump.ku$_status_job_status +
           dbms_datapump.ku$_status_wip, 0, job_state, sts);
js := sts.job_status;
ws := js.worker_status_list;
      dbms_output.put_line('*** Job percent done = ' || to_char(js.percent_done));
      dbms_output.put_line('restarts - '||js.restart_count);
ind := ws.first;
  while ind is not null loop
      dbms_output.put_line( 
           ws(ind).worker_number || ',' 
        || ws(ind).state || ',' 
        || ws(ind).schema || ',' 
        || ws(ind).NAME || ',' 
        || ws(ind).object_type || ',' 
        || ws(ind).degree || ',' 
        || ws(ind).percent_done || ',' 
        || ws(ind).COMPLETED_ROWS
        );

 --   dbms_output.put_line('rows completed - '|| ws(ind));
    ind := ws.next(ind);
  end loop;
  DBMS_DATAPUMP.detach(h1);
  return 1;
exception
  when others then
    if sqlcode = -31626 then return 1;
    else raise;
    end if;
end;
  select job_name , owner_name, get_job ( job_name , owner_name ) 
  from   dba_datapump_jobs;
/

JOB_NAME                OWNER_NAME   GET_JOB(JOB_NAME,OWNER_NAME)   
SYS_IMPORT_SCHEMA_04    CHRIS                                     1 
SYS_IMPORT_TABLE_01     CHRIS                                     1 
SYS_IMPORT_SCHEMA_11    CHRIS                                     1 
SYS_IMPORT_TABLE_04     CHRIS                                     1 
IMPORT_JOB_SQLDEV_206   CHRIS                                     1 
SYS_IMPORT_TABLE_05     CHRIS                                     1 
SYS_IMPORT_TABLE_03     CHRIS                                     1 
SYS_IMPORT_TABLE_06     CHRIS                                     1 
SYS_IMPORT_SCHEMA_10    CHRIS                                     1

Rating

  (7 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

But I want to return all fields from the worker

Mette Stephansen, March 01, 2018 - 2:10 pm UTC

But I want to return all fields from the worker

And I guess that I dont want to call the inline function more than once - so I would like to return a table of varchar2 ie

Instance ID: 1
Instance name: sctest
Host name: B3867S01
Process Name: DW00
State: EXECUTING
Object Schema: SCDAT
Object Name: R_PFCKEYRATIOS_DATASRCIK
Object Type: DATABASE_EXPORT/SCHEMA/TABLE/INDEX/INDEX
Completed Objects: 19,545
Worker Parallelism: 1

I would like to combine it with the first SQL thus +1 line for wach worker

regards Mette

Chris Saxon
March 01, 2018 - 2:21 pm UTC

What's the problem with calling it more than once? You need to pass in the job name and owner for each, right?

Anyway, if you want to return many values, you could create a type and return that:

create or replace type typ as object (
  x int,
  y int
);
/

with function f return typ as
  retval typ;
begin
  retval := typ(1,1);
  
  return retval;
end;
  select f from dual
/

F(X, Y)
-----------------
TYP(1, 1)

I'm not able to create objects (typer)

Mette Stephansen, March 01, 2018 - 2:28 pm UTC

OK - but then I have to have 5-6 inline functions ?
I would think that I stressed Datapump doing a lot of attach_jobs ?

Then I prefer to return a string with all values in a readable format

Regards Mette
Chris Saxon
March 01, 2018 - 5:34 pm UTC

If you can't create objects, then yes, bundling everything into a string is probably the way to go.

But why do you need 5-6 inline functions? You call the same function once for each job, right?

I'm lost - please clarify.

Thanks

Mette Stephansen, March 01, 2018 - 5:51 pm UTC

I'll go with the Return as a string :-)

And then I of course dont need the call of the inline function more than once.

Can I count on the worker number beeing then number as in the name from dba_resumeable ... Onwer.name.worker number ? Or is there another smart way of getting the Worker number ?

Regards Mette
Chris Saxon
March 02, 2018 - 11:15 am UTC

I'm not sure what you're getting at - please clarify. Ideally with an example!

Like this .--

mette Stephansen, March 02, 2018 - 2:37 pm UTC

apart from the fact the is does no work ...
No matter if I type the jobname in a litteral I get and erro from the ATTACH (job not found)

with 
FUNCTION get_worker_status ( 
  p_job_name varchar2, P_owner_name varchar2, p_worker number 
) RETURN varchar2 
AS
  ind NUMBER;              
  h1 NUMBER;               
  percent_done NUMBER;     
  job_state VARCHAR2(30);  
  js ku$_JobStatus;        
  ws ku$_WorkerStatusList; 
  sts ku$_Status;  
  w_result varchar2(300) := 'No Workers found';        

BEGIN

dbms_output.put_line('Here we go ... ' || p_job_name || '-' ||p_owner_name || '-' || p_worker);

h1 := DBMS_DATAPUMP.attach(p_job_name, p_owner_name);
    dbms_output.put_line('We dont here at all');
    
dbms_datapump.get_status(h1,
           dbms_datapump.ku$_status_job_error +
           dbms_datapump.ku$_status_job_status +
           dbms_datapump.ku$_status_wip, 0, job_state, sts
);

js := sts.job_status;
ws := js.worker_status_list;
      dbms_output.put_line('*** Job percent done = ' || to_char(js.percent_done));
      dbms_output.put_line('restarts - '||js.restart_count);
ind := ws.first;

while ind is not null loop
    dbms_output.put_line(ws(ind).worker_number);
    if ws(ind).worker_number = p_worker then
         w_result := 
               ws(ind).worker_number || ',' 
            || ws(ind).state || ',' 
            || ws(ind).schema || ',' 
            || ws(ind).NAME || ',' 
            || ws(ind).object_type || ',' 
            || ws(ind).degree || ',' 
            || ws(ind).percent_done || ',' 
            || ws(ind).COMPLETED_ROWS
        ;
       end if;

    ind := ws.next(ind);
end loop;

  DBMS_DATAPUMP.detach(h1);

  return w_result;
/*  
exception
  when others then
    if sqlcode = -31626 then return 'SQL code - 31626 ?';
    else raise;
    end if; */
end;
select ss.sid
, get_worker_status(s.job_name, s.owner_name, to_number(substr(r.name,instr(r.name,'.',-1)+1,2))) worker_status
, to_number(substr(r.name,instr(r.name,'.',-1)+1,2)) worker
, s.job_name
, j.state
, to_char(ss.logon_time,'dd/mm HH24:mi:ss') job_started
--, j.job_mode
--, j.degree
, dbms_rcvman.Sec2DisplayTime(l.time_remaining) remain
, round((l.sofar/l.totalwork)*100,2) "done%"
--, s.owner_name
, sq.sql_text
, l.message
--, l.totalwork
--, l.sofar
--, round((l.sofar/l.totalwork)*100,2) done
--, l.sql_id
, ss.event
, ss.state
, ss.seconds_in_wait
from dba_datapump_jobs j
left outer join dba_datapump_sessions s on (s.job_name = j.job_name)
left outer join v$session ss            on (ss.saddr = s.saddr)
left outer join v$sql sq                on (ss.sql_id = sq.sql_id)
left outer join v$session_longops l     on (l.sql_id = ss.sql_id)
left outer join dba_resumable r         on (r.session_id = ss.sid)
WHERE 1=1
and ss.module='Data Pump Worker'
--and l.time_remaining > 0
order by job_name, sid, l.start_time
;

Chris Saxon
March 02, 2018 - 4:35 pm UTC

What status are the jobs in? Are they running?

Running ...

mette Stephansen, March 02, 2018 - 5:05 pm UTC

yes the job was running - I could do an attach from cmd line expdp .... attach ..
> Status

And I could see it in the output from the query as well - with "sql code = -31626' in the field worker_status.

Can you attach pictures here somewhere ....

Job: SYS_EXPORT_FULL_01
  Owner: SYSTEM
  Operation: EXPORT
  Creator Privs: TRUE
  GUID: 9607E1863D9842A490A5AEF105B051F4
  Start Time: Friday, 02 March, 2018 17:57:10
  Mode: FULL
  Instance: mette12c
  Max Parallelism: 1
  Timezone: +00:00
  Timezone version: 18
  Endianness: LITTLE
  NLS character set: AL32UTF8
  NLS NCHAR character set: AL16UTF16
  EXPORT Job Parameters:
  Parameter Name      Parameter Value:
     CLIENT_COMMAND        system/******** full=y dumpfile=mette3
     TRACE                 0
  State: EXECUTING
  Bytes Processed: 0
  Current Parallelism: 1
  Job Error Count: 0
  Job heartbeat: 1
  Dump File: C:\APP\MST\ADMIN\METTE12C\DPDUMP\METTE3.DMP
    bytes written: 4,911,104

Worker 1 Status:
  Instance ID: 1
  Instance name: mette12c
  Host name: MIR-165
  Object start time: Friday, 02 March, 2018 17:58:35
  Object status at: Friday, 02 March, 2018 17:58:57
  Process Name: DW00
  State: EXECUTING


regards Mette
Chris Saxon
March 05, 2018 - 4:00 pm UTC

Could it be a permissions issue? System owns the job - which user are you running the SQL command as?

Can you put together a complete test case showing all the steps you took leading to this?

Here is the output from the query in csv format

mette Stephansen, March 02, 2018 - 5:14 pm UTC

SID,WORKER_STATUS,WORKER,JOB_NAME,STATE,JOB_STARTED,REMAIN,done%,SQL_TEXT,MESSAGE,EVENT,STATE_1,SECONDS_IN_WAIT

126,SQL code - 31626 ?,1,SYS_EXPORT_FULL_01,EXECUTING,02/03 18:09:19,,,SELECT /*+all_rows*/ SYS_XMLGEN(VALUE(KU$), XMLFORMAT.createFormat2('TABLE_T', '7')), KU$.OBJ_NUM ,KU$.ANC_OBJ.NAME ,KU$.ANC_OBJ.OWNER_NAME ,KU$.ANC_OBJ.TYPE_NAME ,KU$.BASE_OBJ.NAME ,KU$.BASE_OBJ.OWNER_NAME ,KU$.BASE_OBJ.TYPE_NAME ,KU$.SPARE1 ,KU$.XMLSCHEMACOLS ,KU$.SCHEMA_OBJ.NAME ,KU$.SCHEMA_OBJ.NAME ,'TABLE'............

Logged on as SYS

mette Stephansen, March 05, 2018 - 4:09 pm UTC

I am always logged in as SYS as sysdba

And I can use the anonymous pl/sql to get the wanted information - but not called in a function from SQL

1) export og import something FULL=Y
2) run the SQL (fails to attach job)
3) run the PL/SQL from the original post (attach job goes OK)
4) run export using attach (OK)

Maybe there are PL/SQL packages that cannot be called from a function ?

More to Explore

Data Pump

All of the database utilities including Data Pump are explained in the Utilities guide.