[SATLUG] 10GbE configuration and performance issues
Gregory Alan Hildstrom
hildstrom at hildstrom.com
Wed Jul 18 08:42:49 CDT 2007
Thanks a ton for the lspci tip and other bus bandwidth numbers. I have attached the relevant lspci
entries for the server and client systems below. I looked up the spec sheet on the Intel Memory
Controller Hub (MCH) in the slower client motherboard, which yielded some interesting information.
The motherboard does have 64bit/66MHz pci-x slots, but the total bandwidth between the MCH and the
I/O Controller Hub (ICH or PCI Bridge)) is only 266MB/s ~ 2.12Gb/s, so it is equivalent to a
32bit/66MHz connection. The faster server motherboard has 4GB/s ~ 32Gb/s between the MCH and the
ICH (PCI-e 8x to PCI-X bridge).
http://download.intel.com/design/chipsets/datashts/25252502.pdf
http://www.intel.com/design/chipsets/datashts/30240501.pdf
Thank you all very much for the help. -Greg
--------Server lspci---------
00:00.0 Host bridge: Intel Corporation E7525 Memory Controller Hub (rev 0c)
Subsystem: Intel Corporation E7525 Memory Controller Hub
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Capabilities: [40] Vendor Specific Information
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Bus: primary=00, secondary=06, subordinate=06, sec-latency=32
I/O behind bridge: 00004000-00004fff
Memory behind bridge: d0500000-d05fffff
Prefetchable memory behind bridge: d8000000-dfffffff
Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA+ MAbort- >Reset- FastB2B-
02:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A (rev 09) (prog-if 00
[Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=02, secondary=03, subordinate=03, sec-latency=64
I/O behind bridge: 00002000-00002fff
Memory behind bridge: d0300000-d03fffff
Prefetchable memory behind bridge: 00000000d1000000-00000000d1100000
Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [44] Express PCI/PCI-X Bridge IRQ 0
Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal+ Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 256 bytes, MaxReadReq 512 bytes
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 0
Link: Latency L0s unlimited, L1 unlimited
Link: ASPM Disabled CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x4
Capabilities: [5c] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [6c] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [d8] PCI-X bridge device
Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=100MHz
Status: Dev=02:00.0 64bit- 133MHz- SCD- USC- SCO- SRD-
Upstream: Capacity=65535 CommitmentLimit=65535
Downstream: Capacity=65535 CommitmentLimit=65535
Capabilities: [100] Advanced Error Reporting
Capabilities: [300] Power Budgeting
04:01.0 Ethernet controller: Intel Corporation 82597EX 10GbE Ethernet Controller (rev 02)
Subsystem: Intel Corporation Unknown device a00c
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (63750ns min), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 66
Region 0: Memory at d0400000 (64-bit, non-prefetchable) [size=32K]
Region 2: Memory at d0440000 (64-bit, non-prefetchable) [size=256K]
Region 4: I/O ports at 3000 [size=32]
[virtual] Expansion ROM at d1200000 [disabled] [size=256K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [e4] PCI-X non-bridge device
Command: DPERE- ERO+ RBC=512 OST=3
Status: Dev=04:01.0 64bit+ 133MHz+ SCD- USC- DC=simple DMMRBC=4096 DMOST=3 DMCRS=128 RSCEM-
266MHz- 533MHz-
Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+
Address: 00000000fee00000 Data: 4042
--------End Server lspci-------
--------Client lspci-----------
00:00.0 Host bridge: Intel Corporation 82875P/E7210 Memory Controller Hub (rev 02)
Subsystem: IBM Unknown device 02ae
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
Latency: 0
Region 0: Memory at d2000000 (32-bit, prefetchable) [size=32M]
Capabilities: [e4] Vendor Specific Information
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 0a) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Bus: primary=00, secondary=04, subordinate=04, sec-latency=32
Memory behind bridge: d0300000-d03fffff
Prefetchable memory behind bridge: e0000000-efffffff
Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
03:01.0 Ethernet controller: Intel Corporation Unknown device 109e (rev 02)
Subsystem: Intel Corporation Unknown device a00c
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 52 (63750ns min), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 225
Region 0: Memory at d0200000 (64-bit, non-prefetchable) [size=32K]
Region 2: Memory at d0240000 (64-bit, non-prefetchable) [size=256K]
Region 4: I/O ports at 3000 [size=32]
[virtual] Expansion ROM at 50100000 [disabled] [size=256K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [e4] PCI-X non-bridge device
Command: DPERE- ERO+ RBC=4096 OST=3
Status: Dev=03:01.0 64bit+ 133MHz+ SCD- USC- DC=simple DMMRBC=4096 DMOST=3 DMCRS=128 RSCEM-
266MHz- 533MHz-
Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+
Address: 00000000fee00000 Data: 40e1
--------End Client lspci-------
--- "Daniel J. Givens" <daniel at rugmonster.org> wrote:
> Bruce Dubbs wrote:
> > Gregory Alan Hildstrom wrote:
> >
> >> Hello. I am working on an application that needs 200MB/s sustained
> >> iSCSI throughput for high speed data acquisition.
> >>
> >>
> >
> >
> >> I ran netserver on the server and "netperf -H 192.168.2.151" on the
> >> client. I got 1550Mb/s, but I was expecting something more like
> >> 3-4Gb/s. CPU load on the slower client machine never got above 25%
> >> and I do not know how to measure PCI bus bandwidth utilization.
> >>
> >
> > I don't know how to solve the network issue, but be sure that your disk
> > drives can handle that much data or that you can buffer the data in RAM.
> >
> He's writing to a ramdisk, so there is no disk i/o overhead getting in
> the way.
>
> It sounds to me that the PCI bus may be the constraint. If 2128 Mb/s is
> the theoretical max for 32-bit, I would think you're getting close to
> that at 1550Mb/s. With that said, though, you should be running at
> 64-bit. Have you checked the output of lspci -vv to see if you can get
> any more info on what the card is running at and what latency is there,
> etc? I'm a little stumped too, given what you've said already.
>
> One final thought. Are there any settings in the BIOS that might be
> worth checking out? I haven't been into any of our server's BIOSes
> recently, so I have forgotten what all is in there. Oh yeah, once you
> get it figured out, you're going to have to let us know what the fix
> action was.
>
> Good luck!
> Daniel
> --
> _______________________________________________
> SATLUG mailing list
> SATLUG at satlug.org
> http://alamo.satlug.org/mailman/listinfo/satlug to unsubscribe
> Powered by Rackspace (www.rackspace.com)
>
More information about the SATLUG
mailing list