Well it looks like not having tools installed in one of the guests is the problem. I have a FreeNas VM that is running all the time and I never installed the tools. If that is powered down and two other VMs are running I can switch without issue.
I guess now to actually solve the problem I need to install tools in FreeBSD.
Thanks
