Ug, that's quite a SQL query!
A refcursor can return rows "as soon as they're ready". So if you're waiting a long time it's because that's how long the SQL takes to make the rows available.
I'd look at tuning your SQL. To do this you need an execution plan. For instructions on how to get one read:
https://blogs.oracle.com/sql/entry/how_to_create_an_execution If you need help interpreting the results then post your findings here.
Also: You say you're fetching the results one-by-one in Java. So you've got slow-by-slow processing. I'd look into changing this to use bulk processing. And see if you can run the whole thing in PL/SQL ;)