Although this paper on performance comparison’s of NFS and iSCSI is about a year old, it should give the average reader (who is most probably a sysadmin looking for answers the above protocols) considerable information to start looking intelligently.
The topic needs a more recent treatment since iSCSI has advanced and vendor support is ever increasing. May be someone did and I am drastically out of touch. (either that or my google search terms are deplorable).
I run nfs and I pay a lot of price to do so, as evident by the graphs below.
More memory on the NFS clients, some tweaking to the NFS options (actimeo) helped us cut down the overhead calls. Our environment is also very symlink greedy which contributes to the unusually high number of GETATTR calls. Even after the attrs are cached on the client, GETATTRs are issued to revalidate that information.
With regards to the graphs – your guess is right. They are made using RRD tool. You may want to look at drraw for help with graphs.
In most cases I’ve seen, nfs will be faster then unaccerlated iSCSI, that is to say iscsi without the benefit of a hardware initiator. The Linux and Solaris NFS client has been tuned for a long time to be a pretty efficent data moving machine, so your tests may suprise you.
In this case, you have a pretty good amount of lookup and access calls, which are harder on NFS then their local FS equivalent. But they are still overhead, you may benefit from a directory structure rehash, if you can afford it. It may help you cut down on LOOKUP calls. If you increase the attribute cache, it may cut down your ACCESS calls. You seem to have a low amount of WRITE so a longer cache of attributes should be safe.
My question, is how did you make these wonderful graphs!? I always endup with some ugly awk hack that pipes to a CSV file which I then make a graph in excel with. RRD is much prettier.
-Blake