Hi Narodnik,
The 40 GByte virtual disk is definitely large enough to install Snow Leopard Server... it only needs a few GBytes.
Fusion's support for physical optical drives can be finicky sometimes, depending on the model of drive installed on your system and on the precise behavior of the host OS and guest OS, so it might be worth trying to install from a disc image to see if that's the problem here.
To create a disc image from your Snow Leopard Server media:
On the host, launch Disk Utility. Insert the Snow Leopard Server disc into the drive. In the left pane of Disk Utility, select the optical drive, which will have a name like SuperDrive or perhaps a manufacturer/model number like HL-DT-ST DVD/RW– although it might be tempting to do so, you should not select the name of the optical media itself (Install Snow Leopard Server or something like that) which will appear just below that... It is very important to select the CD/DVD drive and not the inserted media, or you'll get a disc image that will get stuck at the Apple logo.
Choose File> New Disc Image from HL-DT-ST DVD/RW... (or whatever name yours has), and go through that procedure to create a read-only disc image. Save the image file somewhere convenient on the host system. When that procedure is done (it'll take a few minutes), you can eject the Snow Leopard Server disc and put it away somewhere safe.
Now, launch VMware Fusion, and drag the disc image you created into the New Virtual Machine Assistant. Make sure that it's set up to create a Mac OS X Server 10.6 64-bit virtual machine, and continue to create the VM and install the OS, hopefully with more success than your earlier attempts!
Cheers,
--
Darius