Raised a ticket with VMWare and its being worked on. Logs and stats all been submitted in the vmware support portal. This exception is rare but hit us 3 times in last one month. One CacheServer behind one Locator where all our errors surfaced. Environments where we have redundant servers running behind multiple locators never reported this problem. Relevant part of the cache configuration xml below.
----------------client-cache.xml---------------------------
<client-cache>
<pool name="myPool" subscription-enabled="true" socket-buffer-size="256000" read-timeout="120000" retry-attempts="5">
<locator host="myhost.com" port="64444"/>
</pool>
</client-cache>
----------------server-cache.xml---------------------------
<cache>
<gateway-hub id="USUAT" port="${us.execution.gport}" maximum-time-between-pings="120000"/>
<cache-server port="0" maximum-time-between-pings="120000" socket-buffer-size="256000">
<client-subscription eviction-policy="entry" capacity="230000"/>
</cache-server>
</cache>