@mfelker, once agin you're missing the point! The OP wants to test Windows 8 in a VM before installing it on the Host and VMware Workstation is rebooting the Host as he said "my whole system reboots" so your memory comments, especially "Get more memory and post againi." are not really that applicable to begin with in this use case scenario for the moment without knowing the actual technical details and I used 512 MB strictly as an example and have run Windows 8 in a VM for testing purposes, not production, with as low as that and higher values as well. AnthonyL25 certainly can adjust appropriate RAM values based on how much Host RAM is free with a clean boot to Windows XP and then assign a realistic value with what's available to test with! 
After all for all we know the extent of his testing may just be wanting to see the interface first hand and examine a few things to make the decision to install it on the Host. We just don't know since he hasn't delineated what testing for him means and more significantly that is not really as important as finding out what the issue is that is causing the host to reboot and correcting it!
That said, the real issues here is to try and determine whether of not there is an issue with VMware Workstation itself on the Windows XP Host and try and isolate whether or not the issue caries over with other Guest OSes not just Windows 8. Therefore, with the lack of diagnostic information presented thus far in the OP, my suggestions are quite sound and logical from a diagnostic/troubleshooting perspective and that includes collecting/reviewing support information if he'd like to present it!