Thursday, January 12, 2006

Turbocharging an NFS server

When out-of-the box S10/x86 showed mediocre NFS performance (measured by the SPEC SFS benchmark) the following measures improved the results more than twice :
1.increase the dnlc size (set ncsize = 0x100000 in /etc/system)
2.increase the number of concurrent NFS requests (NFSD_SERVERS=128 1024 in /etc/default/nfs)
3.increase the number of VxFS inode table (yes, I am using VxFS)
4.increase the FC HBA queue depth to 256 commands
The dual AMD Opteron 252 box reached 11200 ops/sec with response time 2.4 msec with 24 processes on one load generator.
Kudos to Cyril who helped me to tune the machine.
Solaris10 Solarisx86

7 comments:

ux-admin said...

Could You please detail how you increased the CTQ for the FC adapter(s)?

Thank You in advance!

napobo3 said...

x-posted to:
http://www.opensolaris.org/jive/thread.jspa?forumID=26&threadID=5078
For LSI Logic HBA - in /kernel/drv/itmpt.conf:
max-queue-depth=256;

Emulex HBA with SFK emlx driver doesn't need this. It does 233 queues (weird number, huh...)

napobo3 said...

233 jobs queue, I mean

Anonymous said...

Interesting results... So Solaris gets to
over 11K NFS OPS, how would Linux and FreeBSD perform on such box ?

napobo3 said...

Frankly speaking, Solaris with ZFS instead of VxFS+SVM sits higher in my todo list than Linux and FreeBSD. But it will be interesting to test them (as well as Windoze) too.

napobo3 said...

Spencer wrote:
>Did you attempt to increase the number of load generating processes on the client?

I increased the number of load generating processes on the client to 384 and got 24626 IOps

napobo3 said...

Bob wrote :
From: "palowoda" palowoda@fiver.net
Mailing-List: list solarisx86@yahoogroups.com
Date: Thu, 09 Feb 2006 06:56:31 -0000
Subject: [solarisx86] Interesting blog on Turbocharging NFS on Solaris Opteron


See:
http://napobo3.blogspot.com/2006/01/turbocharging-nfs-server.html

---Bob