[SATLUG] 10GbE configuration and performance issues

[David Power]

Hmmm, I wonder if the driver has to be told to enable TCP hardware
acceleration. 
Using a ram disk makes it sound like a nic card bus or nic to switch issue.
Could one side of the link be half duplex and one full duplex. 
I've had the autoconfig catch me on that and get pretty horrible throughput.

Have you tried a packet dump to see if you are getting tcp back offs?

David

> -----Original Message-----
> From: satlug-bounces at satlug.org [mailto:satlug-bounces at satlug.org] On
> Behalf Of Gregory Alan Hildstrom
> Sent: Tuesday, July 17, 2007 8:29 PM
> To: The San Antonio Linux User's Group Mailing List
> Subject: Re: [SATLUG] 10GbE configuration and performance issues
> 
> I have MTU set to 9000 on the NICs and on the switch ports. Txqueuelen is
> 10000. The netperf tests
> never touch the storage. My iSCSI tests are to 1GB /dev/shm/flatfile
> (ramdisk) block devices for
> now. There should not be any I/O contention on the PCI bus; there is no
> disk I/O and no other
> network I/O during the testing. It's got me pretty stumped at the moment.
> 
> Thanks. -Greg
> 
> --- John Pappas <j at jvpappas.net> wrote:
> 
> > Assume that you have Jumbo Packets configured on the NICs and Switch?
> > You may be hitting a PPS issue on the switch rather than a bus
> > issue...
> >
> > On 7/17/07, David Power <dpower at hal-pc.org> wrote:
> > > What's the drive array on the server look like .
> > > Form the Intel stats page I noticed that the throughput looks more
> like a
> > > 33/66 mhz 32 bit slot.
> > > Is there something sharing that buss that's pulling it down?
> > >
> > > >From the Intel site on the 10Gbe CX4
> > >
> > >          PCI & PCI-X Bus Theoretical Bi-Directional Bus Throughput
> > >
> > >         Bus and Frequency       32-Bit Transfer Rate    64-Bit
> Transfer Rate
> > >         33-MHz PCI              1,064 Mb/sec            2,128 Mb/sec
> > >         66-MHz PCI              2,128 Mb/sec            4,256 Mb/sec
> > >         100-MHz PCI-X           Not applicable          6,400 Mb/sec
> > >         133-MHz PCI-X           Not applicable          8,192 Mb/sec
> > >
> > >
> > > > Hello. I am working on an application that needs 200MB/s sustained
> iSCSI
> > > > throughput for high speed
> > > > data acquisition.
> > > >
> > > > Server:
> > > > IBM x226
> > > > 2x3GHz Xeon
> > > > 4GB RAM
> > > > Intel 10GbE CX4 in PCI-X 100MHz slot (6.4Gb/s max)
> > > > RHEL5
> > > >
> > > > Client:
> > > > IBM x206
> > > > 1x3GHz P4
> > > > 1GB RAM
> > > > Intel 10GbE CX4 in PCI-X 66MHz slot (4.2Gb/s max)
> > > > Fedora Core 6
> > > >
> > > > Network:
> > > > Dell 6224 switch with 2 10GbE modules
> > > >
> > > > I ran netserver on the server and "netperf -H 192.168.2.151" on the
> > > > client. I got 1550Mb/s, but I
> > > > was expecting something more like 3-4Gb/s. CPU load on the slower
> client
> > > > machine never got above
> > > > 25% and I do not know how to measure PCI bus bandwidth utilization.
> > > >
> > > > I tried Myricom's and Intel's tuning suggestions in
> /etc/sysctl.conf, but
> > > > the performance was
> > > > about the same with both sets of settings. Here are Myricom's tuning
> > > > parameters:
> > > > net.core.rmem_max = 16777216
> > > > net.core.wmem_max = 16777216
> > > > net.ipv4.tcp_rmem = 4096 87380 16777216
> > > > net.ipv4.tcp_wmem = 4096 65536 16777216
> > > > net.core.netdev_max_backlog = 250000
> > > > net.ipv4.tcp_timestamps = 0
> > > >
> > > > Any suggestions? Thanks a ton. -Greg
> > > >
> > > > Gregory Alan Hildstrom
> > > > Secure Systems Engineer - Trusted Computer Solutions
> > > > ghildstrom at trustedcs.com
> > > > Software Engineer - Hildstrom Engineering
> > > > hildstrom at hildstrom.com
> > > > Mobile:(210)413-6082
> > > > Home/Fax:(210)599-0469
> > > > Office:(210)340-3151x117
> > > > --
> > > > _______________________________________________
> > > > SATLUG mailing list
> > > > SATLUG at satlug.org
> > > > http://alamo.satlug.org/mailman/listinfo/satlug to unsubscribe
> > > > Powered by Rackspace (www.rackspace.com)
> > >
> > > --
> > > _______________________________________________
> > > SATLUG mailing list
> > > SATLUG at satlug.org
> > > http://alamo.satlug.org/mailman/listinfo/satlug to unsubscribe
> > > Powered by Rackspace (www.rackspace.com)
> > >
> > --
> > _______________________________________________
> > SATLUG mailing list
> > SATLUG at satlug.org
> > http://alamo.satlug.org/mailman/listinfo/satlug to unsubscribe
> > Powered by Rackspace (www.rackspace.com)
> >
> 
> --
> _______________________________________________
> SATLUG mailing list
> SATLUG at satlug.org
> http://alamo.satlug.org/mailman/listinfo/satlug to unsubscribe
> Powered by Rackspace (www.rackspace.com)