Need Advice for Tuning NFS to place nice with a Netapp

[N. Harrington]

Hi
I have several web servers that are attached to a
Netapp (network appliance) unit via NFS-3. A few
servers are 5.5 and a few are 6.1 for comparison
testing. All seem to have lousy performance. When
going through the issues with Netapp, the reasons
given were that we have too many GettAdr/Lookup
requests compared to actual reads. So all the NFS IOPS
are being used up by these requests. As soon as the
webservers get busy, requests pile up.

I have tried everything I can think of. The web
servers are even mounted read only with no help.

My current mount options are:
filer:/vol/fvol31 /home/13/13 nfs
ro,noatime,-r=32768,-T,-b,-R0,-i,-D2,-L 0 0

I have tried plenty of others.

an example nfsstat -c 2 (while working ok)
GtAttr Lookup Rdlink Read Write Rename Access
Rddir
4357 2976 0 125 0 0 3425 0
4173 2836 0 115 0 0 3288 0
4254 2912 0 106 0 0 3344 0
3668 2528 0 99 0 0 2880 0
3992 2746 0 101 0 0 3136 0
3916 2706 0 93 0 0 3080 0
3748 2551 0 106 0 0 2948 0
4121 2851 0 86 0 0 3231 0

Any advice for sysctl tunes or anything else would be
much appreciatted!

Thanks

Nicole



The Large Print Giveth And The Small Print Taketh Away
-- Anon

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

[Jeremy Kister]

Post by N. Harrington
filer:/vol/fvol31 /home/13/13 nfs
ro,noatime,-r=32768,-T,-b,-R0,-i,-D2,-L 0 0

for the same performance reason, mine is:
netapp1:/vol/vol0/export /export nfs rw,-r=16384,-w=16384,-L 0 0

might be worth a try to add the -w.

[N. Harrington]

Post by Jeremy Kister

Post by N. Harrington
filer:/vol/fvol31 /home/13/13 nfs
ro,noatime,-r=32768,-T,-b,-R0,-i,-D2,-L 0 0

netapp1:/vol/vol0/export /export nfs
rw,-r=16384,-w=16384,-L 0 0
might be worth a try to add the -w.
Jeremy Kister
http://jeremy.kister.net./

Hi Jeremy

Thanks for the suggestion. Would using the -w do
anything for me since I am mounting read only?

I will try 16384 again however, as they could not
settle on wether 16k or 32k would be best.


Thanks

Nicole


The Large Print Giveth And The Small Print Taketh Away
-- Anon

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

[Freminlins]

Nicole,

Post by N. Harrington
Hi
I have several web servers that are attached to a
Netapp (network appliance) unit via NFS-3. A few
servers are 5.5 and a few are 6.1 for comparison
testing. All seem to have lousy performance.

We have a similar setup and it runs smoothly.

Can you define "lousy performance" ?

Can you give more details on your network? Are you using Gig ethernet? And
over what medium?

Can you also try just copying a 100MB file from the filer to one of the web
servers and record the time?

Are you running nfsiod?

When

Post by N. Harrington
going through the issues with Netapp, the reasons
given were that we have too many GettAdr/Lookup
requests compared to actual reads. So all the NFS IOPS
are being used up by these requests. As soon as the
webservers get busy, requests pile up.
I have tried everything I can think of. The web
servers are even mounted read only with no help.
filer:/vol/fvol31 /home/13/13 nfs
ro,noatime,-r=32768,-T,-b,-R0,-i,-D2,-L 0 0

Mounting noatime for web servers is a good idea but... your "noatime" option
has no effect on NFS mounts (check out the mount man page). You need "vol
options no_atime_update" on the NetApp.

Any advice for sysctl tunes or anything else would be

Post by N. Harrington
much appreciatted!
Thanks
Nicole

One last thing - are you female?! In a UNIX newsgroup?!

Frem.

[N. Harrington]

Post by Freminlins
Nicole,

Post by N. Harrington
Hi
I have several web servers that are attached to a
Netapp (network appliance) unit via NFS-3. A few
servers are 5.5 and a few are 6.1 for comparison
testing. All seem to have lousy performance.

We have a similar setup and it runs smoothly.

Cool! Can you share with me what sort of settings you
use on your boxes? sysctl/kerneltunes/mount options?

It has taken me a over a month to even get to speak
to someone high enough up he food chain at Netapp to
not say "FreeBSD - that's a version of Linux right?"

Post by Freminlins
Can you define "lousy performance" ?

The web server replies (using either Apache and
Lighthttpd) seem to max out at about 17mb/s. Response
time for the web server will rise gradually, then
suddenly become 10-20seconds for a reply. Much like a
backup on a highway. They claim that the netapp unit
is spending too much time dealing with file
information IOPS than actual transfer of files.
However even on a non in-use server, if I make a
request for a file, that "heavy file access" seems
normal.
IE:
GtAttr Lookup Rdlink Read Write Rename Access
Rddir
0 0 0 0 0 0 0 0
248 160 0 4 0 0 236 0
0 0 0 0 0 0 0 0

Post by Freminlins
Can you give more details on your network? Are you
using Gig ethernet? And
over what medium?

Yes, 4X GigE from the filer via a Vif and trunking on
the switch. A nice 10Gb ready HP unit. I have asked if
using the Vif and trunking could have any effects but
been assured it should not. It does mean I cannot use
jumbo frames. But since web pages and images are
small, I don't think there would be any benefit.

Post by Freminlins
Can you also try just copying a 100MB file from the
filer to one of the web
servers and record the time?

9907187 bytes/sec for a 16M file.
It will transfer in nanotime. So, I believe that
eliminates network performance as an issue.

Post by Freminlins
Are you running nfsiod?

Yes, I show 4 instances running.

Post by Freminlins
When

Post by N. Harrington
going through the issues with Netapp, the reasons
given were that we have too many GettAdr/Lookup
requests compared to actual reads. So all the NFS

IOPS

Post by N. Harrington
are being used up by these requests. As soon as

the

Post by N. Harrington
webservers get busy, requests pile up.
I have tried everything I can think of. The web
servers are even mounted read only with no help.
filer:/vol/fvol31 /home/13/13 nfs
ro,noatime,-r=32768,-T,-b,-R0,-i,-D2,-L 0 0

Mounting noatime for web servers is a good idea
but... your "noatime" option
has no effect on NFS mounts (check out the mount man
page). You need "vol
options no_atime_update" on the NetApp.

Hmm. Drat. We have some web servers that do nothing
but send out data, but some that are used for
uploading and file manipulation. I will have to make
sure that global of an option will not effect what
they do.

Post by Freminlins
Any advice for sysctl tunes or anything else would
be

Post by N. Harrington
much appreciatted!
Thanks
Nicole

One last thing - are you female?! In a UNIX
newsgroup?!

Yup :)
Oh, and yes, I do play the drums :)

Post by Freminlins
Frem.

Thanks for your assistance!!

Nicole



The Large Print Giveth And The Small Print Taketh Away
-- Anon

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

[Freminlins]

On 02/08/06, N. Harrington <***@yahoo.com> wrote:

Cool! Can you share with me what sort of settings you

Post by N. Harrington
use on your boxes? sysctl/kerneltunes/mount options?

This may be a disappointment to you but... I didn't have to do anything :-(
All I have is rw on the client.

It has taken me a over a month to even get to speak

Post by N. Harrington
to someone high enough up he food chain at Netapp to
not say "FreeBSD - that's a version of Linux right?"

It depends who you speak to. There are people at NetApp who know about
FreeBSD.

The web server replies (using either Apache and

Post by N. Harrington
Lighthttpd) seem to max out at about 17mb/s. Response
time for the web server will rise gradually, then
suddenly become 10-20seconds for a reply. Much like a
backup on a highway. They claim that the netapp unit
is spending too much time dealing with file
information IOPS than actual transfer of files.
However even on a non in-use server, if I make a
request for a file, that "heavy file access" seems
normal.
GtAttr Lookup Rdlink Read Write Rename Access
Rddir
0 0 0 0 0 0 0 0
248 160 0 4 0 0 236 0
0 0 0 0 0 0 0 0

I would dispute NetApp's claim. NetApp filers are very capable at doing NFS
operations. Static files tend to sit usefully in the buffer cache on web
servers. So unless you are doing something really odd with your web servers
I would tend to disagree with NetApp.

I've just looked on one of our POP3 servers (mounting NetApps). POP3 causes
far more random access than our web servers. As such it doesn't sit in the
buffer cache very long.

We have much higher figures than yours and absolutely no performance
problems.

Yes, 4X GigE from the filer via a Vif and trunking on

Post by N. Harrington
the switch. A nice 10Gb ready HP unit. I have asked if
using the Vif and trunking could have any effects but
been assured it should not. It does mean I cannot use
jumbo frames. But since web pages and images are
small, I don't think there would be any benefit.

There is nothing wrong in theory with that setup. But is may not be what you
want. Try it with just one GigE interface.

You're right - you probably don't need jumbo frames.

Which interface does the HP unit have? Also, have a look at netstat -in. Are
there any IErrs or OErrs or Coll? Paste the results here!

9907187 bytes/sec for a 16M file.

Post by N. Harrington
It will transfer in nanotime. So, I believe that
eliminates network performance as an issue.

Well, not really. The figure above is showing < 10MB a second. That's not
quite Fast Ethernet speed. If you are pushing 17mb (I guess that's megabits)
that's not really a problem though.

I've just tested this on the same POP3 server above, using dd to write a
file onto a NetApp and I get 10889359 a second. And this machine is busy.
Also, it is mounting the NetApp over Fast Ethernet.

Hmm. Drat. We have some web servers that do nothing

Post by N. Harrington
but send out data, but some that are used for
uploading and file manipulation. I will have to make
sure that global of an option will not effect what
they do.

It is a per volume option. And frankly I've never seen much use for atime.
It's useful sometimes, but not a lot.

Can you also put in the output of nfsstat -W -c 2. Maybe it's best to put
this up on the web somewhere as it's wide, and it's not easy to read in
email. Let it run for a minute or so, and if possible do two runs. One
during the OK time, the other during the problem time.

I would go back to basics. One GigE interface. Just rw mount options, and
start testing. By testing I mean measuring. NFS tuning is fiddly. I've been
using NetApps with FreeBSD for 5 years. It is a good combination.

Can you also post the output of sysctl -a|grep nfs. But don't start fiddling
with them yet!

Post by N. Harrington
One last thing - are you female?! In a UNIX

Post by Freminlins
newsgroup?!

Yup :)
Oh, and yes, I do play the drums :)

Oh gawd. Whatever next? :-)

Thanks for your assistance!!

Post by N. Harrington
Nicole

Frem.