Skip to Main Content
  • Questions
  • How to replace sqlloader control file with external table

Breadcrumb

Question and Answer

Connor McDonald

Thanks for the question, Michael.

Asked: October 16, 2017 - 8:23 am UTC

Last updated: October 16, 2017 - 2:12 pm UTC

Version: 11g

Viewed 1000+ times

You Asked

Hello,

I have a flat file structured like this:

HH00001 0000000048JZFRV288SNGAA814
DD11111111 desc1 000000000000000000012 100002
DD02324234 desc2 000001200000000000000 00 300012 100002
DD54534564 desc3 000000111000000000000 23 0511 100003
HH00002 0000000049JZF3YD2SG4FAAA23
DD23233554 desc5 000000200000000000000 00 23Y6 100002
DD01111111 desc6 000000200000000000000 00 0002 00002

> HH - header
> DD - detail

I need to populate two tables (head and detail) creating a relationship..
so I'm expecting 2 head records, and 5 details all related (sort of FK)
The TYPERECORD, which are the first 2 chars, marks the type(HH o DD)

Actually I'm using sql loader

OPTIONS (ROWS=1)          
    LOAD DATA
    INFILE 'file.txt'
    BADFILE 'file.bad'
    DISCARDFILE 'file.dsc'
    --TRUNCATE
    APPEND
    into table TABLE_H
      WHEN P_TYPERECORD = 'HH'
      TRAILING NULLCOLS 
      (  
          TABLE_H_ID                  expression "SQ_TABLE_H.NEXTVAL",
       P_TYPERECORD                POSITION(1:2),
       P_NUM                       POSITION(3:7),
       P_OPTIONAL                  POSITION(8:14),
       P_DESC                      POSITION(15:40)
      )
    into table TABLE_D
      WHEN P_TYPERECORD = 'DD'
      TRAILING NULLCOLS 
     ( 
         TABLE_D_ID            expression "SQ_TABLE_D.NEXTVAL", 
          TABLE_H_ID         expression "SQ_TABLE_H.CURRVAL",
       P_TYPERECORD                POSITION(1:2),
       P_DESC                      POSITION(3:10),
       P_OPTIONAL                  POSITION(11:18),
       P_DESC                      POSITION(19:46),
          ...
       
     )


How can I implement the same way via external table?
I don't know how can I use the sequence in order to make this sort of relationship

and Chris said...

SQL*Loader has a handy external_table option. Pass generate only to this and you'll get the script to create and load your external table in the logfile:

C:\Users\csaxon\Documents\Scripts>type sqlldr.ctl
OPTIONS (ROWS=1)
    LOAD DATA
    INFILE 'file.txt'
    BADFILE 'file.bad'
    DISCARDFILE 'file.dsc'
    --TRUNCATE
    APPEND
    into table TABLE_H
      WHEN P_TYPERECORD = 'HH'
      TRAILING NULLCOLS
      (
          TABLE_H_ID                  expression "SQ_TABLE_H.NEXTVAL",
       P_TYPERECORD                POSITION(1:2),
       P_NUM                       POSITION(3:7),
       P_OPTIONAL                  POSITION(8:14),
       P_DESC                      POSITION(15:40)
      )
    into table TABLE_D
      WHEN P_TYPERECORD = 'DD'
      TRAILING NULLCOLS
     (
         TABLE_D_ID            expression "SQ_TABLE_D.NEXTVAL",
          TABLE_H_ID         expression "SQ_TABLE_H.CURRVAL",
       P_TYPERECORD                POSITION(1:2),
       P_DESC                      POSITION(3:10),
       P_OPTIONAL                  POSITION(11:18)
     )
C:\Users\csaxon\Documents\Scripts>sqlldr userid=chris/chris@db control=sqlldr.ctl external_table=generate_only

SQL*Loader: Release 12.2.0.1.0 - Production on Mon Oct 16 12:13:53 2017

Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.

Path used:      External Table
SQL*Loader-800: ROWS specified and ignored

C:\Users\csaxon\Documents\Scripts>type sqlldr.log

SQL*Loader: Release 12.2.0.1.0 - Production on Mon Oct 16 12:13:53 2017

Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.

Control File:   sqlldr.ctl
Data File:      file.txt
  Bad File:     file.bad
  Discard File: file.dsc
 (Allow all discards)

Number to load: ALL
Number to skip: 0
Errors allowed: 50
Continuation:    none specified
Path used:      External Table

Table TABLE_H, loaded when P_TYPERECORD = 0X4848(character 'HH')
Insert option in effect for this table: APPEND
TRAILING NULLCOLS option in effect

   Column Name                  Position   Len  Term Encl Datatype
------------------------------ ---------- ----- ---- ---- ---------------------
TABLE_H_ID                                                EXPRESSION
    SQL string for column : "SQ_TABLE_H.NEXTVAL"
P_TYPERECORD                          1:2     2           CHARACTER
P_NUM                                 3:7     5           CHARACTER
P_OPTIONAL                           8:14     7           CHARACTER
P_DESC                              15:40    26           CHARACTER

Table TABLE_D, loaded when P_TYPERECORD = 0X4444(character 'DD')
Insert option in effect for this table: APPEND
TRAILING NULLCOLS option in effect

   Column Name                  Position   Len  Term Encl Datatype
------------------------------ ---------- ----- ---- ---- ---------------------
TABLE_D_ID                                                EXPRESSION
    SQL string for column : "SQ_TABLE_D.NEXTVAL"
TABLE_H_ID                                                EXPRESSION
    SQL string for column : "SQ_TABLE_H.CURRVAL"
P_TYPERECORD                          1:2     2           CHARACTER
P_DESC                               3:10     8           CHARACTER
P_OPTIONAL                          11:18     8           CHARACTER

SQL*Loader-800: ROWS specified and ignored


CREATE DIRECTORY statements needed for files
------------------------------------------------------------------------
CREATE DIRECTORY SYS_SQLLDR_XT_TMPDIR_00000 AS 'C:\Users\csaxon\Documents\Scripts'


CREATE TABLE statement for external table:
------------------------------------------------------------------------
CREATE TABLE "SYS_SQLLDR_X_EXT_TABLE_H"
(
  "X_000_001" NUMBER(38),
  "X_000_002" NUMBER(38),
  "X_000_003" NUMBER(38),
  "X_000_004" NUMBER(38),
  "X_001_002" NUMBER(38),
  "X_001_003" NUMBER(38),
  "X_001_004" NUMBER(38)
)
ORGANIZATION external
(
  TYPE oracle_loader
  DEFAULT DIRECTORY SYS_SQLLDR_XT_TMPDIR_00000
  ACCESS PARAMETERS
  (
    RECORDS DELIMITED BY NEWLINE CHARACTERSET WE8MSWIN1252
    BADFILE 'SYS_SQLLDR_XT_TMPDIR_00000':'file.bad'
    DISCARDFILE 'SYS_SQLLDR_XT_TMPDIR_00000':'file.dsc'
    LOGFILE 'sqlldr.log_xt'
    READSIZE 1048576
    FIELDS LDRTRIM
    MISSING FIELD VALUES ARE NULL
    REJECT ROWS WITH ALL NULL FIELDS
    (
      "X_000_001" (1:2) CHAR(2),
      "X_000_002" (3:7) CHAR(5),
      "X_000_003" (8:14) CHAR(7),
      "X_000_004" (15:40) CHAR(26),
      "X_001_002" (1:2) CHAR(2),
      "X_001_003" (3:10) CHAR(8),
      "X_001_004" (11:18) CHAR(8)
    )
  )
  location
  (
    'file.txt'
  )
)REJECT LIMIT UNLIMITED


CREATE TABLE for internal table used in multi-table load:
------------------------------------------------------------------------
CREATE TABLE "SYS_SQLLDR_X_INT_TABLE_H"
(
  "X_000_001" NUMBER(38),
  "X_000_002" NUMBER(38),
  "X_000_003" NUMBER(38),
  "X_000_004" NUMBER(38),
  "X_001_002" NUMBER(38),
  "X_001_003" NUMBER(38),
  "X_001_004" NUMBER(38)
)


INSERT statement used to load internal table (multi-table load):
------------------------------------------------------------------------
INSERT /*+ append */ INTO "SYS_SQLLDR_X_INT_TABLE_H"
(
  "X_000_001",
  "X_000_002",
  "X_000_003",
  "X_000_004",
  "X_001_002",
  "X_001_003",
  "X_001_004"
)
SELECT
  "X_000_001",
  "X_000_002",
  "X_000_003",
  "X_000_004",
  "X_001_002",
  "X_001_003",
  "X_001_004"
FROM "SYS_SQLLDR_X_EXT_TABLE_H"


INSERT statements used to load internal tables:
------------------------------------------------------------------------
INSERT /*+ append */ INTO TABLE_H
(
  TABLE_H_ID,
  P_TYPERECORD,
  P_NUM,
  P_OPTIONAL,
  P_DESC
)
SELECT
  SQ_TABLE_H.NEXTVAL,
  "X_000_001",
  "X_000_002",
  "X_000_003",
  "X_000_004"
FROM "SYS_SQLLDR_X_INT_TABLE_H"
WHERE ("X_000_001" = 'HH')

INSERT /*+ append */ INTO TABLE_D
(
  TABLE_D_ID,
  TABLE_H_ID,
  P_TYPERECORD,
  P_DESC,
  P_OPTIONAL
)
SELECT
  SQ_TABLE_D.NEXTVAL,
  SQ_TABLE_H.CURRVAL,
  "X_001_002",
  "X_001_003",
  "X_001_004"
FROM "SYS_SQLLDR_X_INT_TABLE_H"
WHERE ("X_001_002" = 'DD')


statements to cleanup objects created by previous statements:
------------------------------------------------------------------------
DROP TABLE "SYS_SQLLDR_X_INT_TABLE_H"
DROP TABLE "SYS_SQLLDR_X_EXT_TABLE_H"
DROP DIRECTORY SYS_SQLLDR_XT_TMPDIR_00000



Run began on Mon Oct 16 12:13:53 2017
Run ended on Mon Oct 16 12:13:59 2017

Elapsed time was:     00:00:05.93
CPU time was:         00:00:00.33


Note you don't place sequences in the external table definition. You call them in your insert script.

Unless you're on 12.2+. In which case you can use sequence defaults in the table definition:

create sequence s;
create table t (
  x int default s.nextval
) organization external (
  default directory tmp
  location ('test.txt')
);

Rating

  (1 rating)

Is this answer out of date? If it is, please let us know via a Comment

Comments

using External tables

Rajeshwaran Jeyabal, October 16, 2017 - 1:03 pm UTC

into table TABLE_D
WHEN P_TYPERECORD = 'DD'
TRAILING NULLCOLS 
( 
 TABLE_D_ID            expression "SQ_TABLE_D.NEXTVAL", 
  TABLE_H_ID         expression "SQ_TABLE_H.CURRVAL",
P_TYPERECORD                POSITION(1:2),
P_DESC                      POSITION(3:10),


The most important about this question is how to maitain relationship across tables using constraints and sequences, while loading from external tables.

let's start with this data.

DACCOUNTING ,NEW YORK
ECLARK      ,MANAGER
EMILLER     ,CLERK
EKING       ,PRESIDENT
DRESEARCH   ,DALLAS
ESMITH      ,CLERK
EFORD       ,ANALYST
EADAMS      ,CLERK
ESCOTT      ,ANALYST
EJONES      ,MANAGER
DSALES      ,CHICAGO
EALLEN      ,SALESMAN
EJAMES      ,CLERK
ETURNER     ,SALESMAN
EBLAKE      ,MANAGER
EMARTIN     ,SALESMAN
EWARD       ,SALESMAN
DOPERATIONS ,BOSTON

create table DEPT(
 deptno number ,
 dname varchar2(20),
 location varchar2(20),
 constraint dept_pk primary key(deptno) );
 
create table EMP(
 empno number,
 ename varchar2(25),
 deptno number, 
  constraint emp_pk primary key(empno) ,
  constraint emp_fk foreign key(deptno)
   references dept ); 

create sequence dept_seq;
create sequence emp_seq;


I have the above delimited data,
a) if the record starts with 'D' then it should go into DEPT table and have a Sequence generated PK.
b) using that sequence generated PK from DEPT table load all the record starts with 'E' into EMP table

so the final output after the load should be like this.

demo@ORA12C> select * from dept;

    DEPTNO DNAME          LOC
---------- -------------- -------------
        1 ACCOUNTING     NEW YORK
        2 RESEARCH       DALLAS
        3 SALES          CHICAGO

demo@ORA12C> select empno,ename,deptno from emp order by deptno;

     EMPNO ENAME          DEPTNO
---------- ---------- ----------
      7782 CLARK              1
      7839 KING               1
      7934 MILLER             1
      7566 JONES              2
      7902 FORD               2
      7876 ADAMS              2
      7369 SMITH              2
      7788 SCOTT              2
      7521 WARD               3
      7844 TURNER             3
      7499 ALLEN              3
      7900 JAMES              3
      7698 BLAKE              3
      7654 MARTIN             3

14 rows selected.


demo@ORA12C>


Using External Tables - it goes like this.

demo@ORA12C> CREATE TABLE ext_tbl
  2  (
  3    "X" VARCHAR2(20),
  4    "Y" VARCHAR2(30)
  5  )
  6  ORGANIZATION external
  7  (
  8    TYPE oracle_loader
  9    DEFAULT DIRECTORY TMP
 10    ACCESS PARAMETERS
 11    (
 12      RECORDS DELIMITED BY NEWLINE CHARACTERSET WE8MSWIN1252
 13      BADFILE 'TMP':'ctl.bad'
 14      LOGFILE 'log.txt_xt'
 15      READSIZE 1048576
 16      FIELDS TERMINATED BY "," LDRTRIM
 17      REJECT ROWS WITH ALL NULL FIELDS
 18      (
 19        "X" CHAR(255)
 20          TERMINATED BY ",",
 21        "Y" CHAR(255)
 22          TERMINATED BY ","
 23      )
 24    )
 25    location
 26    (
 27      'data.dat'
 28    )
 29  )REJECT LIMIT UNLIMITED ;

Table EXT_TBL created.

demo@ORA12C> select * from ext_tbl;

X                    Y
-------------------- ------------------------------
DACCOUNTING          NEW YORK
ECLARK               MANAGER
EMILLER              CLERK
EKING                PRESIDENT
DRESEARCH            DALLAS
ESMITH               CLERK
EFORD                ANALYST
EADAMS               CLERK
ESCOTT               ANALYST
EJONES               MANAGER
DSALES               CHICAGO
EALLEN               SALESMAN
EJAMES               CLERK
ETURNER              SALESMAN
EBLAKE               MANAGER
EMARTIN              SALESMAN
EWARD                SALESMAN
DOPERATIONS          BOSTON

18 rows selected.

demo@ORA12C> alter table emp disable constraint emp_fk;

Table altered.

demo@ORA12C> alter table emp disable constraint emp_pk;

Table altered.

demo@ORA12C> alter table dept disable constraint dept_pk;

Table altered.

demo@ORA12C> insert all
  2     when substr(x,1,1) ='D' then
  3             into dept(deptno,dname,location) values(dept_seq.nextval,names#,values#)
  4     when substr(x,1,1)='E' then
  5             into emp(empno,ename,deptno) values(emp_seq.nextval,names#,dept_seq.currval)
  6  select x, substr(x,2) names#, y values#
  7  from ext_tbl;

18 rows created.

demo@ORA12C> select * from dept;

    DEPTNO DNAME                LOCATION
---------- -------------------- --------------------
         1 ACCOUNTING           NEW YORK
         5 RESEARCH             DALLAS
        11 SALES                CHICAGO
        18 OPERATIONS           BOSTON

demo@ORA12C> select * from emp ;

     EMPNO ENAME                         DEPTNO
---------- ------------------------- ----------
         2 CLARK                              2
         3 MILLER                             3
         4 KING                               4
         6 SMITH                              6
         7 FORD                               7
         8 ADAMS                              8
         9 SCOTT                              9
        10 JONES                             10
        12 ALLEN                             12
        13 JAMES                             13
        14 TURNER                            14
        15 BLAKE                             15
        16 MARTIN                            16
        17 WARD                              17

14 rows selected.

demo@ORA12C> 



Questions:
1) How do we guarantee the query against the External table ( select * from external_table ) will read the data sequentially from the data file? if it reads the data randomly then the above requirement can be handled using External Tables?
Connor McDonald
October 16, 2017 - 2:12 pm UTC

I think we can infer this from

a) the docs, each location is a granule

"External tables treats each data file specified on the LOCATION clause as a single granule"

b) Bug 6812439

(now fixed) which was locations were not being read in the correct order

Or to be safe, you could use RECNUM in the definition and subsequently order by that.

eg

https://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:52733181746448#52977916329285

More to Explore

Utilities

All of the database utilities are explained in the Utilities guide.