Security Server uses the JVM too.
At least 10GB is recommended for Connection Server (including replica Connection Server and Security Server). See table 1-1 on page 8 here http://pubs.vmware.com/view-51/topic/com.vmware.ICbase/PDF/view-51-installation.pdf
Mark