Folks,
This issues is not resolved but have a workaround which led to a big leap in performance but still not what it should be. Using VMware IOAnalyzer appliance a default ESXi 5.1 host was get 2.3MB/s on a 512k blk 100%read, 0% random test to an NFS mount over a 1gb link. After much testing we found if we disable the rx Interrupt Moderation we could get 74MB over the same link. This is the commands we used to modify the Interrupt Moderation:
| ethtool -C vmnic5 rx-usecs 0 rx-frames 1 rx-usecs-irq 0 rx-frames-irq 0 | |
| ethtool -C vmnic4 rx-usecs 0 rx-frames 1 rx-usecs-irq 0 rx-frames-irq 0 |
http://www.odbms.org/download/vmw-vfabric-gemFire-best-practices-guide.pdf
Hope this helps someone else.
Terry