this query is wronger than wrong:
select /*+ INDEX_ASC(records recpk) */ * from records where ((system_id='123' and cust_id='3456218791') and rec_no>'0000' ) and rownum<2 order by system_id asc,cust_id asc,rec_no asc;
it is giving CORRECT information on all nodes because what is says to do is:
find one random records that match the predicate and then sort it.
You want to get the data matching your predicate, sort them and then get the first two.
IT HAS TO BE:
select * from (
select /*+ first_rows(2) */ * from records where ((system_id='123' and cust_id='3456218791') and rec_no>'0000' )
order by system_id asc,cust_id asc,rec_no asc)
where rownum < 2;
that is the ONLY way to write that query semantically correctly.
Use that and only that query. Find the developer that wrote the bad query and educate them on how rownum works. This is called a top-n query and can only be written using rownum with an inline view. There are no shortcuts, you have to use the inline view to do a top-n query - period.
http://docs.oracle.com/cd/E11882_01/server.112/e17118/pseudocolumns009.htm#SQLRF00255 You cannot rely on a hint being observed - they are called HINTS, they are not directives. If the index is unavailable for any reason (different name, unusable, whatever), then the query will just ignore it. If it were a directive, the query would fail, but it won't. You have to code it the way I've demonstrated and I would recommend not using the INDEX_ASC hint.
ugh, and you've used strings to store all kinds of numbers :(