[SATLUG] 10GbE configuration and performance issues

[David Power]

Never mind

> -----Original Message-----
> From: satlug-bounces at satlug.org [mailto:satlug-bounces at satlug.org] On
> Behalf Of David Power
> Sent: Wednesday, July 18, 2007 9:24 AM
> To: 'The San Antonio Linux User's Group Mailing List'
> Subject: RE: [SATLUG] 10GbE configuration and performance issues
> 
> Hmmm, I wonder if the driver has to be told to enable TCP hardware
> acceleration.
> Using a ram disk makes it sound like a nic card bus or nic to switch
> issue.
> Could one side of the link be half duplex and one full duplex.
> I've had the autoconfig catch me on that and get pretty horrible
> throughput.
> 
> Have you tried a packet dump to see if you are getting tcp back offs?
> 
> David
> 
> > -----Original Message-----
> > From: satlug-bounces at satlug.org [mailto:satlug-bounces at satlug.org] On
> > Behalf Of Gregory Alan Hildstrom
> > Sent: Tuesday, July 17, 2007 8:29 PM
> > To: The San Antonio Linux User's Group Mailing List
> > Subject: Re: [SATLUG] 10GbE configuration and performance issues
> >
> > I have MTU set to 9000 on the NICs and on the switch ports. Txqueuelen
> is
> > 10000. The netperf tests
> > never touch the storage. My iSCSI tests are to 1GB /dev/shm/flatfile
> > (ramdisk) block devices for
> > now. There should not be any I/O contention on the PCI bus; there is no
> > disk I/O and no other
> > network I/O during the testing. It's got me pretty stumped at the
> moment.
> >
> > Thanks. -Greg
> >
> > --- John Pappas <j at jvpappas.net> wrote:
> >
> > > Assume that you have Jumbo Packets configured on the NICs and Switch?
> > > You may be hitting a PPS issue on the switch rather than a bus
> > > issue...
> > >
> > > On 7/17/07, David Power <dpower at hal-pc.org> wrote:
> > > > What's the drive array on the server look like .
> > > > Form the Intel stats page I noticed that the throughput looks more
> > like a
> > > > 33/66 mhz 32 bit slot.
> > > > Is there something sharing that buss that's pulling it down?
> > > >
> > > > >From the Intel site on the 10Gbe CX4
> > > >
> > > >          PCI & PCI-X Bus Theoretical Bi-Directional Bus Throughput
> > > >
> > > >         Bus and Frequency       32-Bit Transfer Rate    64-Bit
> > Transfer Rate
> > > >         33-MHz PCI              1,064 Mb/sec            2,128 Mb/sec
> > > >         66-MHz PCI              2,128 Mb/sec            4,256 Mb/sec
> > > >         100-MHz PCI-X           Not applicable          6,400 Mb/sec
> > > >         133-MHz PCI-X           Not applicable          8,192 Mb/sec
> > > >
> > > >
> > > > > Hello. I am working on an application that needs 200MB/s sustained
> > iSCSI
> > > > > throughput for high speed
> > > > > data acquisition.
> > > > >
> > > > > Server:
> > > > > IBM x226
> > > > > 2x3GHz Xeon
> > > > > 4GB RAM
> > > > > Intel 10GbE CX4 in PCI-X 100MHz slot (6.4Gb/s max)
> > > > > RHEL5
> > > > >
> > > > > Client:
> > > > > IBM x206
> > > > > 1x3GHz P4
> > > > > 1GB RAM
> > > > > Intel 10GbE CX4 in PCI-X 66MHz slot (4.2Gb/s max)
> > > > > Fedora Core 6
> > > > >
> > > > > Network:
> > > > > Dell 6224 switch with 2 10GbE modules
> > > > >
> > > > > I ran netserver on the server and "netperf -H 192.168.2.151" on
> the
> > > > > client. I got 1550Mb/s, but I
> > > > > was expecting something more like 3-4Gb/s. CPU load on the slower
> > client
> > > > > machine never got above
> > > > > 25% and I do not know how to measure PCI bus bandwidth
> utilization.
> > > > >
> > > > > I tried Myricom's and Intel's tuning suggestions in
> > /etc/sysctl.conf, but
> > > > > the performance was
> > > > > about the same with both sets of settings. Here are Myricom's
> tuning
> > > > > parameters:
> > > > > net.core.rmem_max = 16777216
> > > > > net.core.wmem_max = 16777216
> > > > > net.ipv4.tcp_rmem = 4096 87380 16777216
> > > > > net.ipv4.tcp_wmem = 4096 65536 16777216
> > > > > net.core.netdev_max_backlog = 250000
> > > > > net.ipv4.tcp_timestamps = 0
> > > > >
> > > > > Any suggestions? Thanks a ton. -Greg
> > > > >
> > > > > Gregory Alan Hildstrom
> > > > > Secure Systems Engineer - Trusted Computer Solutions
> > > > > ghildstrom at trustedcs.com
> > > > > Software Engineer - Hildstrom Engineering
> > > > > hildstrom at hildstrom.com
> > > > > Mobile:(210)413-6082
> > > > > Home/Fax:(210)599-0469
> > > > > Office:(210)340-3151x117
> > > > > --
> > > > > _______________________________________________
> > > > > SATLUG mailing list
> > > > > SATLUG at satlug.org
> > > > > http://alamo.satlug.org/mailman/listinfo/satlug to unsubscribe
> > > > > Powered by Rackspace (www.rackspace.com)
> > > >
> > > > --
> > > > _______________________________________________
> > > > SATLUG mailing list
> > > > SATLUG at satlug.org
> > > > http://alamo.satlug.org/mailman/listinfo/satlug to unsubscribe
> > > > Powered by Rackspace (www.rackspace.com)
> > > >
> > > --
> > > _______________________________________________
> > > SATLUG mailing list
> > > SATLUG at satlug.org
> > > http://alamo.satlug.org/mailman/listinfo/satlug to unsubscribe
> > > Powered by Rackspace (www.rackspace.com)
> > >
> >
> > --
> > _______________________________________________
> > SATLUG mailing list
> > SATLUG at satlug.org
> > http://alamo.satlug.org/mailman/listinfo/satlug to unsubscribe
> > Powered by Rackspace (www.rackspace.com)
> 
> --
> _______________________________________________
> SATLUG mailing list
> SATLUG at satlug.org
> http://alamo.satlug.org/mailman/listinfo/satlug to unsubscribe
> Powered by Rackspace (www.rackspace.com)