regarding to your queries
For a normal datastore the queue depth on a HBA is 32, - The datastore queue depth is default 32 in esxi.esx, this can be increased but..normally not recommeded.
The queue depth can be modified using the Disk.SchedNumReqOutstanding advanced parameter. Values between 1 and 256 (concurrent commands) are acceptable. With out proper analysis it is not recomended to change this.
does an RDM for example have it's own queue in the same way as a datastore ?, sure it is also have ...
is there also queueing on the virtual machine in a software HBA, - in software hba the queueing will be in the LUN level no HBA level.
is there any limit on the number of RDM's that are supported on a host? in a datacenter? - the default limit of esx is 256 luns so this also applies to RDM
Also when dealing RDM these below facts is also useful
VM with Physical (Pass-Thru) RDMs (Powered On – Storage vMotion):
- If I try to change the format to thin or thick, then no Storage vMotion allowed.
- If I chose not to do any conversion, only the pRDM mapping file is moved from the source VMFS datastore to the destination VMFS datastore – the data stays on the original LUN.
VM with Virtual (non Pass-Thru) RDMs (Power On – Storage vMotion):
- On a migrate, if I chose to covert the format in the advanced view, the vRDM is converted to a VMDK on the destination VMFS datastore.
- If I chose not to do any conversion, only the vRDM mapping file is moved from the source VMFS datastore to the destination VMFS datastore – the data stays on the original LUN (same behaviour as pRDM)
VM with Physical (Pass-Thru) RDMs (Powered Off – Cold Migration):
- On a migrate, if I chose to change the format (via the advanced view), the pRDM is converted to a VMDK on the destination VMFS datastore.
- If I chose not to do any conversion, only the pRDM mapping file is moved from the source VMFS datastore to the destination VMFS datastore – the data stays on the original LUN
VM with Virtual (non Pass-Thru) RDMs (Power Off – Cold Migration):
- On a migrate, if I chose to covert the format in the advanced view, the vRDM is converted to a VMDK on the destination VMFS datastore.
- If I chose not to do any conversion, only the vRDM mapping file is moved from the source VMFS datastore to the destination VMFS datastore – the data stays on the original LUN (same behaviour as pRDM).