Skip to Main Content
  • Questions
  • Extract only "executable" SELECT code block within a very long SQL Text string that contains comments

Breadcrumb

Question and Answer

Connor McDonald

Thanks for the question, Sameer.

Asked: March 15, 2017 - 12:35 am UTC

Last updated: April 08, 2017 - 5:28 am UTC

Version: 11.2.0.4 Enterprise

Viewed 1000+ times

You Asked

Hello,

I have a column of type CLOB in a table, where a user could copy-paste their entire "SQL Query Text" string so that a procedure will later "process" the entries in this table and use DBMS_SQL to parse out the query string to see if it is valid or not and if valid, it will go ahead and use DBMS_SQL to execute it as well.

As part of the procedure's validation checks, I would like to ensure the following:-

1. Skip any commenting code blocks which start with '/*' and end with '*/' anywhere it appears in the entire SQL Query Text.

2. Skip any commenting code lines which start with '--' anywhere in the entire SQL Query Text.

3. After skipping/ignoring through 1 & 2 above, ensure that the first "executable" statement MUST start with the word 'SELECT'. Any other words like INSERT or DROP or DELETE or UPDATE or anything else is not accepted and the procedure shall display a friendly message to the user as well as RAISE an exception.

Note: Please keep in mind that a user could also include the word 'SELECT' within a commenting code block (either between /* and */ or as part of --lines, so we really need to find an "executable" SELECT instead of any word 'SELECT' appearing anywhere.

4. Once I find the first "executable" statement is indeed a 'SELECT' word, I will proceed with replacing it with a customized SELECT string containing additional compiler hints concatenated. Example: Replace 'SELECT c1, c2.....' with 'SELECT /*+ PARALLEL(n) NOLOGGING */ c1, c2...'

5. Once the final string is assembled, proceed with executing the query using either DBMS_SQL.EXECUTE or EXECUTE IMMEDIATE (not sure which one would be appropriate and why?)


This table will be used to dynamically "process" for creating TABLES (using CTAS only), VIEWS and MATERIALIZED VIEWS as they all require a SELECT query to populate these objects.

Any help would be appreciated.

Thanks in advance!

and Connor said...

Something like this should get you started

SQL> create table t (c clob);

Table created.

SQL>
SQL> declare
  2    l_sql clob :=
  3  q'{ /* some initial stuff
  4      which we dont care about
  5      -- and comments in comments
  6   */
  7   select * -- all cols
  8   from   user_tables
  9   where  table_name like 'T%'
 10   -- and num_rows > 0
 11   /* omitting these ones
 12   and nested = 'Y'
 13   and user_stats = 'Y'
 14   */ order by 1}';
 15  begin
 16    insert into t values (l_sql );
 17    commit;
 18  end;
 19  /

PL/SQL procedure successfully completed.

SQL>
SQL> select * from t;

C
---------------------------------------------------------------------------------------------------
--------
 /* some initial stuff
    which we dont care about
    -- and comments in comments
 */
 select * -- all cols
 from   user_tables
 where  table_name like 'T%'
 -- and num_rows > 0
 /* omitting these ones
 and nested = 'Y'
 and user_stats = 'Y'
 */ order by 1


1 row selected.

SQL>
SQL> set serverout on
SQL> declare
  2    l_text clob;
  3    l_clean clob;
  4    l_in_comment   boolean := false;
  5    l_in_comment_mode varchar2(10);
  6    pos pls_integer;
  7  begin
  8    for i in ( select * from t )
  9    loop
 10      l_text  := ltrim(i.c);
 11      l_clean := null;
 12      pos := 1;
 13      loop
 14        if substr(l_text,pos,2) = '--'
 15        then
 16          if l_in_comment then
 17            null;
 18          else
 19            l_in_comment := true;
 20            l_in_comment_mode := 'dashes';
 21          end if;
 22          pos := pos + 2;
 23        elsif substr(l_text,pos,2) = '/*'
 24        then
 25          if l_in_comment then
 26            null;
 27          else
 28            l_in_comment := true;
 29            l_in_comment_mode := 'slashes';
 30          end if;
 31          pos := pos + 2;
 32        elsif substr(l_text,pos,1) = chr(10)
 33        then
 34          if l_in_comment and l_in_comment_mode = 'dashes' then
 35            l_in_comment := false;
 36          else
 37            l_clean := l_clean || substr(l_text,pos,1);
 38          end if;
 39          pos := pos + 1;
 40        elsif substr(l_text,pos,2) = '*/'
 41        then
 42          if l_in_comment and l_in_comment_mode = 'slashes' then
 43            l_in_comment := false;
 44          else
 45            l_clean := l_clean || substr(l_text,pos,2);
 46          end if;
 47          pos := pos + 2;
 48        else
 49          if not l_in_comment then
 50            l_clean := l_clean || substr(l_text,pos,1);
 51          end if;
 52          pos := pos + 1;
 53        end if;
 54        exit when pos > length(l_text);
 55      end loop;
 56    end loop;
 57    dbms_output.put_line(l_clean);
 58  end;
 59  /




 select *  from   user_tables
 where  table_name like 'T%'



 order by 1

PL/SQL procedure successfully completed.




But I must stress, *anything* that takes SQL text from a source and then might run it needs to be REALLY REALLY REALLY REALLY tightly controlled because the risks are huge in terms of security.

Anyway, once you have a cleansed sql statement, could do something like this to check its validity:

SQL> declare
  2    l_sql clob := 'select * from tab';
  3    l_cursor number := dbms_sql.open_cursor;
  4  begin
  5    if substr(ltrim(upper(l_sql),' '||chr(13)||chr(10)),1,7) = 'SELECT ' then
  6      dbms_sql.parse( l_cursor,l_sql, dbms_sql.native );
  7    else
  8      raise_application_error(-20000,'SELECT not the leading keyword');
  9    end if;
 10  exception
 11    when others then
 12      dbms_sql.close_cursor( l_cursor );
 13      raise;
 14  end;
 15  /

PL/SQL procedure successfully completed.


If you are expecting lots of SQL's, then you probably dont want to smoke your shared pool, so you might something like:

SQL> declare
  2    l_sql clob := 'select * from tab';
  3    l_cursor number := dbms_sql.open_cursor;
  4  begin
  5    if substr(ltrim(upper(l_sql),' '||chr(13)||chr(10)),1,7) = 'SELECT ' then
  6      execute immediate 'alter session set cursor_sharing=force';
  7      dbms_sql.parse( l_cursor, l_sql, dbms_sql.native );
  8      execute immediate 'alter session set cursor_sharing=exact';
  9    else
 10      raise_application_error(-20000,'SELECT not the leading keyword');
 11    end if;
 12  exception
 13    when others then
 14      execute immediate 'alter session set cursor_sharing=exact';
 15      dbms_sql.close_cursor( l_cursor );
 16      raise;
 17  end;
 18  /

PL/SQL procedure successfully completed.


Rating

  (4 ratings)

Is this answer out of date? If it is, please let us know via a Comment

Comments

or

A reader, March 15, 2017 - 5:27 pm UTC

1- trasnpose the clob as lines
2- use analytics or patterns

eg. start by this :

with
str as
(select replace(c, chr(10), 'chr(10)') as x from t
),
lst as
(select trim(regexp_substr(upper(str.x),'[^chr(10)]+',1,level)) as val
from str, dual
connect by level <= length(regexp_replace(str.x,'[^chr(10)]'))+1
)
select * from lst
;
then for each line use analytics and others funct to remove the lines of comments (.. think Connor and chris are aces of analytics so haapy if they show us the rest)


Thanks!

Sameer Premji, March 15, 2017 - 6:48 pm UTC

Thanks a lot Connor.

Your code definitely has given me a jump-start where I was stuck previously.

The only thing I'd like to add (for other users reading this thread) is that I forgot to add one more condition that if the user copy-pasted the SQL Query Text with an ending ';' terminator symbol, the DBMS_SQL.PARSE throws an error and doesn't like it.

Using your code, I now have a "cleansed and executable SQL statement" with which I could further strip-out the ';' and embed my custom SELECT string.

Thanks once again for this :)
Connor McDonald
March 16, 2017 - 1:24 am UTC

glad we could help

Hit a snag with compiler hints!

A reader, April 07, 2017 - 9:35 pm UTC

Hi Connor,

Your code worked like a charm to strip-off all the comment blocks between '/*' and '*/'as well as comment lines that had '--' characters.

However, I hit a snag today when one of the users had a compiler hint embedded within the SELECT clause like:-

SELECT /*+ APPEND */ c1, c2....

Your code encounters the '/*' symbol and removes the '+ APPEND ' string.

Any chance you could provide a revised code such that it also takes into account NOT to strip-off any line that starts with symbol '/*+' and ends with '*/'.

I tried to modify your code but ended-up messing it up further.

Thanks in advance!
Connor McDonald
April 08, 2017 - 5:28 am UTC

Hmmm.... I sortta reckon you could have worked this out yourself ...

SQL> create table t (c clob);

Table created.

SQL>
SQL> declare
  2    l_sql clob :=
  3  q'{ /* some initial stuff
  4      which we dont care about
  5      -- and comments in comments
  6   */
  7   insert /*+ append */ into xxx
  8   select * -- all cols
  9   from   user_tables
 10   where  table_name like 'T%'
 11   -- and num_rows > 0
 12   /* omitting these ones
 13   and nested = 'Y'
 14   and user_stats = 'Y'
 15   */ order by 1}';
 16  begin
 17    insert into t values (l_sql );
 18    commit;
 19  end;
 20  /

PL/SQL procedure successfully completed.

SQL>
SQL>
SQL>  set serverout on
SQL>  declare
  2     l_text clob;
  3     l_clean clob;
  4     l_in_comment   boolean := false;
  5     l_in_comment_mode varchar2(10);
  6     pos pls_integer;
  7   begin
  8     for i in ( select * from t )
  9     loop
 10       l_text  := ltrim(i.c);
 11       l_clean := null;
 12       pos := 1;
 13       loop
 14         if substr(l_text,pos,2) = '--'  and substr(l_text,pos,3) != '--+'
 15         then
 16           if l_in_comment then
 17             null;
 18           else
 19             l_in_comment := true;
 20             l_in_comment_mode := 'dashes';
 21           end if;
 22           pos := pos + 2;
 23         elsif substr(l_text,pos,2) = '/*'  and substr(l_text,pos,3) != '/*+'
 24         then
 25           if l_in_comment then
 26             null;
 27           else
 28             l_in_comment := true;
 29             l_in_comment_mode := 'slashes';
 30           end if;
 31           pos := pos + 2;
 32         elsif substr(l_text,pos,1) = chr(10)
 33         then
 34           if l_in_comment and l_in_comment_mode = 'dashes' then
 35             l_in_comment := false;
 36           else
 37             l_clean := l_clean || substr(l_text,pos,1);
 38           end if;
 39           pos := pos + 1;
 40         elsif substr(l_text,pos,2) = '*/'
 41         then
 42           if l_in_comment and l_in_comment_mode = 'slashes' then
 43             l_in_comment := false;
 44           else
 45             l_clean := l_clean || substr(l_text,pos,2);
 46           end if;
 47           pos := pos + 2;
 48         else
 49           if not l_in_comment then
 50             l_clean := l_clean || substr(l_text,pos,1);
 51           end if;
 52           pos := pos + 1;
 53         end if;
 54         exit when pos > length(l_text);
 55       end loop;
 56     end loop;
 57     dbms_output.put_line(l_clean);
 58   end;
 59  /




 insert /*+ append */ into xxx
 select *  from   user_tables
 where  table_name like 'T%'



 order by 1


Thank you!

A reader, April 08, 2017 - 6:24 am UTC

Worked like a charm....appreciate it Connor!

More to Explore

CLOB/BLOB/etc

Complete documentation on Securefiles and Large Objects here