Currently voms-proxy-* command calls java without setting the maximum heap size (-Xmx options). This way java choose how many memory to reserve and this may clash with shell limits.

The problem manifested on machine with large memory capacity and shell memory limit (ulimit -v) in place. To reproduce the problem on a machine with normal memory capacity setting a low shell memory limit is needed.
As how much memory java will try to reserve is dependent on the total ram, first determine that by monitoring the memory usage of a java -version call

[root@jen05 ~]# while [ 1 ]; do top -b -n 1 | grep java ; done
5720 root 17 0 2333m 11m 7152 S 6.0 0.1 0:00.03 java

Since java tries to reserve more than 2GB of memory, setting the limit to 1GB makes it fails

[root@jen05 ~]# ulimit -v
unlimited
[root@jen05 ~]# java -version
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.11.90) (rhel-1.41.1.11.11.90.el5_9-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

[root@jen05 ~]# ulimit -v 1000000
[root@jen05 ~]# java -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

To workaround the problem set _JAVA_OPTIONS to a value under the shell limit

[root@jen05 ~]# ulimit -v
2000000
root@jen05 ~]# export _JAVA_OPTIONS=-Xmx512m
[root@jen05 ~]# java -version
Picked up _JAVA_OPTIONS: -Xmx512m
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.11.90) (rhel-1.41.1.11.11.90.el5_9-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

VOMS clients should set a sensible default for Xmx based on their needs, and provide a way to set a different one.