Skip to Main Content
  • Questions
  • Single Json object is splitting into multiple lines when the length is more than 32767.

Breadcrumb

Question and Answer

Connor McDonald

Thanks for the question.

Asked: January 08, 2018 - 5:45 pm UTC

Last updated: January 09, 2018 - 4:19 pm UTC

Version: 12.1.0.2.0

Viewed 1000+ times

You Asked

Hi Team,

Happy New Year.

I am connecting to an External database from unix server with that external database details and executing a select statement to fetch data from one of the table in the
external database and then the output will be written to a file.

Before I execute the select statement I am using the following sqlplus settings.

set linesize 32767
set pagesize 0
set serveroutput on
set arraysize 5000
set long 50000
set LONGCHUNKSIZE 50000

The data in the table is in Json format and after the output file is created with Json data I will use this file to stage into my external table in our database.

But I have seen an issue that when there is any Json string with more than 32767 characters then that entire Json is splitting into next line. Because of this the whole Json becomes
invalid Json format and logging as bad record when I stage the output file into External table.

If I don't use set linesize 32767 then the Json is getting truncated at default linesize.

Since the max linsize is 32767 please suggest how to avoid the Json splitting into two lines when the Json length is more than 32767 characters.

I am not sure how I can provide sample code as I am using external database connections to connect and a simple select statement to fetch the data and output will be written to a file and this file is being stored in our server.

But please help me how can I overcome this issue. I appreciate your inputs.

and Chris said...

So you're reading a JSON document stored in a column, spooling it using SQL*Plus? And these documents can be larger than 32,767 characters, so you're exceeding the maximum linesize?

If so, here's a few things you could try:

Manually add carriage returns

At appropriate points in the document, e.g. after close braces "}", add a carriage return:

with jdoc as (
 select '[{"employee_id":100,"first_name":"Steven"},{"employee_id":101,"first_name":"Neena"},{"employee_id":102,"first_name":"Lex"},{"employee_id":103,"first_name":"Alexander"},{"employee_id":104,"first_name":"Bruce"},{"employee_id":105,"first_name":"David"},{"employee_id":106,"first_name":"Valli"},{"employee_id":107,"first_name":"Diana"},{"employee_id":108,"first_name":"Nancy"},{"employee_id":109,"first_name":"Daniel"}]' j from dual
)
  select replace(j, '}', '}' || chr(10)) 
  from   jdoc;

REPLACE(J,'}','}'||CHR(10))                                                                                                                                                                                                                                                                                                                                                                                                                 
[{"employee_id":100,"first_name":"Steven"}
,{"employee_id":101,"first_name":"Neena"}
,{"employee_id":102,"first_name":"Lex"}
,{"employee_id":103,"first_name":"Alexander"}
,{"employee_id":104,"first_name":"Bruce"}
,{"employee_id":105,"first_name":"David"}
,{"employee_id":106,"first_name":"Valli"}
,{"employee_id":107,"first_name":"Diana"}
,{"employee_id":108,"first_name":"Nancy"}
,{"employee_id":109,"first_name":"Daniel"}
] 


Combine the lines when reading the file

You could stick with spooling the file as you currently do. And accept that documents will be split over many lines.

Then stitch the lines back together when you read them. Connor has an example of how to do this using pipelined table functions on his blog:

https://connor-mcdonald.com/2017/11/01/parsing-freeform-data-in-flat-files/

Skip the file writing step!

It sounds like your process is:

- Write the file from database A
- Read it into database B as an external table

If so, why the need to create the file? With a database link between the databases, you can transfer in one step:

insert into t@dbb
  select * from t@dba;

Rating

  (1 rating)

Is this answer out of date? If it is, please let us know via a Comment

Comments

A reader, January 23, 2018 - 5:39 pm UTC