depending upon the license we can take different approach, basically the below we can do
With standard switch - and no enterprise license - no ether channel
vSwitch0 - 2 nics - default teaming policy - we can put management and VMotion traffic - use separate VLAN - use multinic vmotion - also you can put some VM traffic also may be some management VM's
moreinfo below for multinic vmotion
http://pibytes.wordpress.com/2013/01/12/multi-nic-vmotion-in-vsphere-5-x/
http://pibytes.wordpress.com/2013/01/12/multi-nic-vmotion-speed-and-performance-in-vsphere-5-x/
vSwitch1 - 2 nics - default teaming policy - for VM traffic - create portgroups etc...
vSwitch2 - 2 nics - default teaming policy - for ISCSI Traffic - make 2 port groups - and in the port groups select the teaming policy and select OVER RIDE switch fail over order - make one nic acitve for 1 iscsi port group and other nic as un used - ESXI iscsi uses multipathing for failover no need for selecting failover in NIC TEAMING
vSwitch3 - 2 Nics - default teaming policy - NFS traffic - make 2 port groups - and in the port groups select the teaming policy and select OVER RIDE switch fail over order - make one nic acitve for 1 iscsi port group and other nic as standby - nfs needs this because the failover is handled by the vswitch
vSwitch3 - 2 nics - you can use this for FT
Now another case - with NICS in ether channel
the cisco 3750 supports MLSA- that is multi chasis ether channel and for that you need to make ether channel and use IP hash teaming policy and not use BECON probing for failover detection this will cause vlan flapping issue so use default.
in this case you can follow the same as above, depeding upon the you environment if NFS traffic is more you can use 4 nics and if iscsi traffic is more you can use 4 nics and its upon you.
you can refer the netapp best practice guide...it has all the details..YES it is LONG...
IF you have more nics... you can give 4 nics each for NFS and ISCSI that is 2 nics to each switch and make a ether channel with 4 nics across 2 pswitches..
IF you have enterprise plus license.... you can use Net IOC, here you can club all the nics and spread across to 2 switch and make a big etherchannel and use one distributed vswitch.. and or no need to use etherchannel... we can use LBT that is good...then use net ioc to set shares and bandwidth allocation...
similar thread in the community - http://communities.vmware.com/message/2081040#2081040
So there is lot.... of ways to do...