Skip to Main Content
  • Questions
  • array processing on dynamic data loading method 4

Breadcrumb

Question and Answer

Tom Kyte

Thanks for the question, Miquel .

Asked: February 13, 2005 - 1:14 pm UTC

Last updated: July 30, 2019 - 3:23 am UTC

Version: 9.0.2

Viewed 1000+ times

You Asked

Hi Tom,
i use a function in PL/SQL , like the 'load_data' function of your book Expert one-on-one , to load a uknown number of fields from a flat file to a unknown table at compile-time ,with DBMS_SQL ( a variation to handle multiple record-types and multiple tables, but that's not relevant here ).
Like you say in that chapter this kind of routine is adequate for small amount of data, but now i have larger files to load , and i want to enhance it with array processing (the time to load 6 files
with an average of 7000 lines is about 6 min.).
Something like the DBMSSQL_ARRAY routine on chapter 16, using dbms_sql.bind_array, will help, but here the number of fields is hard-coded :
l_stmt:='insert into '|| p_tname || ' q1 (a,b,c) values (:a, :b, :c)';
And there are three dbms_sql tables declared.If the number of inputs (and columns) are uknown i can't do that.
Can you give some example solution for this (load_data with string column_names parameter and array processing).

Tanks in advance. (Great site & great books!)



and Tom said...

You only need one array -- a table of varchar2's


bind_array takes two subscripts -- the low and high bounds of the array to use for the i'th column. So, say you want to use an arraysize of 100, you can use 1..100 for column 1, 101..200 for column 2, 201..300 for column 3 and so on.


All you need to do is figure out how to build the insert, which you would just do based on the dictionary -- using to_number for the numbers, to_date for the dates with the appropriate format, and nothing on the strings.

Taking that very same example:


ops$tkyte@ORA9IR2> create or replace package load_data
  2  as
  3
  4  procedure dbmssql_array( p_tname     in varchar2,
  5                           p_arraysize in number default 100,
  6                           p_rows      in number default 500 );
  7
  8  end load_data;
  9  /
 
Package created.
 
ops$tkyte@ORA9IR2>
ops$tkyte@ORA9IR2> create or replace package body load_data
  2  as
  3
  4  procedure dbmssql_array( p_tname     in varchar2,
  5                           p_arraysize in number default 100,
  6                           p_rows      in number default 500 )
  7  is
  8      l_stmt      long;
  9      l_theCursor integer;
 10      l_status    number;
 11      l_data      dbms_sql.varchar2_table;
 12      l_cnt       number default 0;
 13  begin
 14      l_stmt := 'insert into ' || p_tname ||
 15                ' q1 ( a, b, c ) values ( to_number(:a), to_date(:b,''yyyymmddhh24miss''), :c )';
 16
 17      l_theCursor := dbms_sql.open_cursor;
 18      dbms_sql.parse(l_theCursor, l_stmt, dbms_sql.native);
 19
 20      for i in 1 .. p_rows
 21      loop
 22          l_cnt := l_cnt+1;
 23          l_data( l_cnt+(0*p_arraysize) ) := i;
 24          l_data( l_cnt+(1*p_arraysize) ) := to_char( sysdate+i, 'yyyymmddhh24miss' );
 25          l_data( l_cnt+(2*p_arraysize) ) := to_char(i);
 26
 27          if (l_cnt = p_arraysize)
 28          then
 29             dbms_sql.bind_array( l_theCursor, ':a', l_data, 0*p_arraysize+1, 0*p_arraysize+l_cnt );
 30             dbms_sql.bind_array( l_theCursor, ':b', l_data, 1*p_arraysize+1, 1*p_arraysize+l_cnt );
 31             dbms_sql.bind_array( l_theCursor, ':c', l_data, 2*p_arraysize+1, 2*p_arraysize+l_cnt );
 32             l_status := dbms_sql.execute( l_theCursor );
 33             l_cnt := 0;
 34          end if;
 35      end loop;
 36      if (l_cnt > 0 )
 37      then
 38          dbms_sql.bind_array( l_theCursor, ':a', l_data, 0*p_arraysize+1, 0*p_arraysize+l_cnt );
 39          dbms_sql.bind_array( l_theCursor, ':b', l_data, 1*p_arraysize+1, 1*p_arraysize+l_cnt );
 40          dbms_sql.bind_array( l_theCursor, ':c', l_data, 2*p_arraysize+1, 2*p_arraysize+l_cnt );
 41          l_status := dbms_sql.execute( l_theCursor );
 42      end if;
 43      dbms_sql.close_cursor( l_theCursor );
 44  end;
 45
 46  end;
 47  /
 
Package body created.
 
ops$tkyte@ORA9IR2> drop table t;
 
Table dropped.
 
ops$tkyte@ORA9IR2> create table t ( a number, b date, c varchar2(30) );
 
Table created.
 
ops$tkyte@ORA9IR2>
ops$tkyte@ORA9IR2> exec load_data.dbmssql_array( 'T', 10, 15 );
 
PL/SQL procedure successfully completed.
 
ops$tkyte@ORA9IR2> select * from t;
 
         A B         C
---------- --------- ------------------------------
         1 14-FEB-05 1
         2 15-FEB-05 2
         3 16-FEB-05 3
         4 17-FEB-05 4
         5 18-FEB-05 5
         6 19-FEB-05 6
         7 20-FEB-05 7
         8 21-FEB-05 8
         9 22-FEB-05 9
        10 23-FEB-05 10
        11 24-FEB-05 11
        12 25-FEB-05 12
        13 26-FEB-05 13
        14 27-FEB-05 14
        15 28-FEB-05 15
 
15 rows selected.



See how:

dbms_sql.bind_array
( l_theCursor, ':a', l_data, 0*p_arraysize+1, 0*p_arraysize+l_cnt );
dbms_sql.bind_array
( l_theCursor, ':b', l_data, 1*p_arraysize+1, 1*p_arraysize+l_cnt );
dbms_sql.bind_array
( l_theCursor, ':c', l_data, 2*p_arraysize+1, 2*p_arraysize+l_cnt );


lets you bind non-overlapping slices of a single array?

Rating

  (4 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

Great

Muhammad Riaz Shahid, February 14, 2005 - 12:20 am UTC

Hello Tom !

Could you share with us that load_data procedure (or package whatsoever it is) for Dynamic Table & Columns ? Also please confirm that the flat file should be present at server. right? What if the file is on Client ?

Regards


Tom Kyte
February 14, 2005 - 8:28 am UTC

I share everything I have right here.

This is a modification of an example from expert one on one Oracle where I was comparing the performance of native dynamic sql, dbms_sql, single row modifications, array operations.

You need to

a) use the dictionary to figure out the columns and types so you can
b) create the proper insert statement so you can
c) read the OS file, parse each line and populate the arrays so you can
d) call a routine like the above prototyped one.....

if you want the server to read a file, the file needs be on the server.

if you want a client file to be read, you need a program running on the client to read the file.

A little Mistake

Muhammad Riaz Shahid, February 14, 2005 - 12:23 am UTC

Hello !

Please ignore "from Dubai, Pakistan" in my previous post....it should have have been "from Dubai, UAE"

:(



Miquel, February 14, 2005 - 5:16 am UTC

thanks a lot, great solution!

Automation of this function . Do you see any problem?

Miquel, February 21, 2005 - 1:53 pm UTC

Tom, what do you think of having this function called every 'x' minutes , scheduled with several jobs.
I am working on a application who must receive and send text files in order to communicate with other applications (orders from customers, wharehouse ship order,...).
The application scans every 'x' minutes a group of directories for text files, and loads them . (The text files tipically are small, max. 500 lines)
I have tested with a high number of files (3000, the total received per day), and the performance is better than the current code (is static, but poorly writen, and different for each type of text file).
I know 'well written' static code will be perhaps a little faster , but in this case is a lot of code to re-write.
With the dynamic approach, i'll schedule a job for each type of file, but in the end the load function will be the same (yours, the above function).
What do you think of that approach?

Tank you very much.







Tom Kyte
February 21, 2005 - 3:39 pm UTC

seems OK to me -- there are times generic is appropriate -- parsing a delmited file for loading could be one of them.

More to Explore

PL/SQL demos

Check out more PL/SQL tutorials on our LiveSQL tool.

PL/SQL docs

PL/SQL reference manual from the Oracle documentation library