My System Admin already looked into all the kernal parameters - there
is 16GB free memory on that box and 48 GB free swap space.

It is defintely some Oracle "PGA" kind of limitaion (e.g., length of
sql statement or some memory for sql statamenet parsing etc ..)
Since it is third part applicaion, I do not have access to source code.