The following diff needs testing with PCIe bge(4) chipsets.

To give people an idea of which chipsets are PCIe look for
dmesg entries such as the following...

bge0 at pci2 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1
bge0 at pci2 dev 0 function 0 "Broadcom BCM5722" rev 0x00, BCM5755 C0
bge0 at pci1 dev 0 function 0 "Broadcom BCM5751" rev 0x01, BCM5750 A1
bge0 at pci2 dev 0 function 0 "Broadcom BCM5751M" rev 0x11, BCM5750 B1
bge0 at pci4 dev 0 function 0 "Broadcom BCM5752" rev 0x02, BCM5752 A2
bge0 at pci1 dev 0 function 0 "Broadcom BCM5753M" rev 0x21, BCM5750 C1
bge0 at pci8 dev 0 function 0 "Broadcom BCM5754" rev 0x02, BCM5754/5787 A2
bge0 at pci5 dev 0 function 0 "Broadcom BCM5755M" rev 0x02, BCM5755 A2
bge0 at pci3 dev 0 function 0 "Broadcom BCM5787M" rev 0x02, BCM5754/5787 A2
bge0 at pci3 dev 0 function 0 "Broadcom BCM5789" rev 0x11, BCM5750 B1

Please provide me with a dmesg.


Index: if_bge.c
================================================== =================
RCS file: /cvs/src/sys/dev/pci/if_bge.c,v
retrieving revision 1.236
diff -u -p -r1.236 if_bge.c
--- if_bge.c 7 Jun 2008 19:05:11 -0000 1.236
+++ if_bge.c 7 Jul 2008 07:12:36 -0000
@@ -1669,8 +1669,8 @@ bge_blockinit(struct bge_softc *sc)
dma_read_modebits =
BGE_RDMAMODE_ENABLE | BGE_RDMAMODE_ALL_ATTNS;

- if (sc->bge_flags & BGE_PCIE && 0)
- dma_read_modebits |= BGE_RDMA_MODE_FIFO_LONG_BURST;
+ if (sc->bge_flags & BGE_PCIE)
+ dma_read_modebits |= BGE_RDMAMODE_FIFO_LONG_BURST;

CSR_WRITE_4(sc, BGE_RDMA_MODE, dma_read_modebits);
}
Index: if_bgereg.h
================================================== =================
RCS file: /cvs/src/sys/dev/pci/if_bgereg.h,v
retrieving revision 1.83
diff -u -p -r1.83 if_bgereg.h
--- if_bgereg.h 20 Apr 2008 01:32:43 -0000 1.83
+++ if_bgereg.h 7 Jul 2008 07:12:14 -0000
@@ -1362,10 +1362,8 @@
#define BGE_RDMAMODE_PCI_FIFOOREAD_ATTN 0x00000100
#define BGE_RDMAMODE_LOCWRITE_TOOBIG 0x00000200
#define BGE_RDMAMODE_ALL_ATTNS 0x000003FC
-
-/* Alternate encodings for PCI-Express, from Broadcom-supplied Linux driver */
-#define BGE_RDMA_MODE_FIFO_LONG_BURST ((1 << 17) || (1 << 16))
-#define BGE_RDMA_MODE_FIFO_SIZE_128 (1 << 17)
+#define BGE_RDMAMODE_FIFO_SIZE_128 0x00020000
+#define BGE_RDMAMODE_FIFO_LONG_BURST 0x00030000

/* Read DMA status register */
#define BGE_RDMASTAT_PCI_TGT_ABRT_ATTN 0x00000004

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

hi!

it works fine and improves the bge tx performance.

the testing was done from a Thinkpad T61 (client, em0 PCI-E) to the HP
workstation (server, bge0 PCI-E). sorry for using iperf, tcpbench
misses the very useful "-d" dual mode feature for lazy people like me
.

the tested NIC was (see dmesg below):
bge0 at pci3 dev 0 function 0 "Broadcom BCM5752" rev 0x01, BCM5752 A1 (0x6001): irq 10, address 00:15:60:9d:72:d9

reyk

from em0 to bge0 without the diff:

$ sudo iperf -w 128k -i 2 -c 192.168.1.20 -d
Password:
WARNING: option -d is not valid in single threaded versions
------------------------------------------------------------
Client connecting to 192.168.1.20, TCP port 5001
TCP window size: 128 KByte
------------------------------------------------------------
[ 3] local 192.168.1.15 port 42534 connected with 192.168.1.20 port 5001
[ 3] 0.0- 2.0 sec 165 MBytes 691 Mbits/sec
[ 3] 2.0- 4.0 sec 165 MBytes 692 Mbits/sec
[ 3] 4.0- 6.0 sec 170 MBytes 712 Mbits/sec
[ 3] 6.0- 8.0 sec 177 MBytes 741 Mbits/sec
[ 3] 0.0-10.0 sec 853 MBytes 716 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte
------------------------------------------------------------
[ 4] local 192.168.1.15 port 5001 connected with 192.168.1.20 port 4827
[ 4] 0.0- 2.0 sec 118 MBytes 496 Mbits/sec
[ 4] 2.0- 4.0 sec 118 MBytes 496 Mbits/sec
[ 4] 4.0- 6.0 sec 118 MBytes 496 Mbits/sec
[ 4] 6.0- 8.0 sec 117 MBytes 492 Mbits/sec
[ 4] 0.0-10.0 sec 586 MBytes 494 Mbits/sec

from em0 to bge0 with the diff:

$ sudo iperf -w 128k -i 2 -c 192.168.1.20 -d
WARNING: option -d is not valid in single threaded versions
------------------------------------------------------------
Client connecting to 192.168.1.20, TCP port 5001
TCP window size: 128 KByte
------------------------------------------------------------
[ 3] local 192.168.1.15 port 39329 connected with 192.168.1.20 port 5001
[ 3] 0.0- 2.0 sec 165 MBytes 691 Mbits/sec
[ 3] 2.0- 4.0 sec 165 MBytes 693 Mbits/sec
[ 3] 4.0- 6.0 sec 165 MBytes 692 Mbits/sec
[ 3] 6.0- 8.0 sec 172 MBytes 721 Mbits/sec
[ 3] 0.0-10.0 sec 844 MBytes 708 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte
------------------------------------------------------------
[ 4] local 192.168.1.15 port 5001 connected with 192.168.1.20 port 30484
[ 4] 0.0- 2.0 sec 224 MBytes 938 Mbits/sec
[ 4] 2.0- 4.0 sec 224 MBytes 939 Mbits/sec
[ 4] 4.0- 6.0 sec 224 MBytes 941 Mbits/sec
[ 4] 6.0- 8.0 sec 224 MBytes 938 Mbits/sec
[ 4] 0.0-10.0 sec 1.09 GBytes 939 Mbits/sec

OpenBSD 4.4-beta (GENERIC) #0: Fri Jul 11 12:39:38 CEST 2008
root@pardon.hq.vantronix.net:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) 4 CPU 3.00GHz ("GenuineIntel" 686-class) 3 GHz
cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,P GE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE, SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,EST,CNXT-ID,CX16,xTPR
real mem = 527851520 (503MB)
avail mem = 502190080 (478MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 05/18/05, BIOS32 rev. 0 @ 0xeb660, SMBIOS rev. 2.4 @ 0xeeb40 (69 entries)
bios0: vendor Hewlett-Packard version "786D1 v01.03" date 05/18/2005
bios0: Hewlett-Packard HP Compaq dc7600 Convertible Minitower
acpi0 at bios0: rev 0
acpi0: tables DSDT FACP SSDT APIC ASF! MCFG
acpi0: wakeup devices PCI0(S4) PEG1(S4) PCX1(S4) PCX2(S4) PCX5(S4) PCX6(S4) HUB_(S4) COM1(S4) COM2(S4) USB1(S1) USB2(S1) USB3(S1) USB4(S1) EUSB(S1) PBTN(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEG1)
acpiprt2 at acpi0: bus 32 (PCX1)
acpiprt3 at acpi0: bus 63 (PCX2)
acpiprt4 at acpi0: bus -1 (PCX5)
acpiprt5 at acpi0: bus -1 (PCX6)
acpiprt6 at acpi0: bus 5 (HUB_)
acpicpu0 at acpi0
acpibtn0 at acpi0: PBTN
bios0: ROM list: 0xc0000/0xaa00! 0xcaa00/0x1000 0xcba00/0x1800 0xe8c00/0x7400!
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel 82945G Host" rev 0x02
ppb0 at pci0 dev 1 function 0 "Intel 82945G PCIE" rev 0x02: irq 11
pci1 at ppb0 bus 1
thtc0 at pci1 dev 0 function 0 "Tehuti Networks TN3014" rev 0x00: irq 11
tht0 at thtc0 port 0: address 00:1b:56:18:00:99
tht1 at thtc0 port 1: address 00:1b:56:18:80:99
vga1 at pci0 dev 2 function 0 "Intel 82945G Video" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
agp0 at vga1: aperture at 0xd0000000, size 0x10000000
azalia0 at pci0 dev 27 function 0 "Intel 82801GB HD Audio" rev 0x01: irq 11
azalia0: codec[s]: Realtek ALC260
audio0 at azalia0
ppb1 at pci0 dev 28 function 0 "Intel 82801GB PCIE" rev 0x01
pci2 at ppb1 bus 32
ppb2 at pci0 dev 28 function 1 "Intel 82801GB PCIE" rev 0x01: irq 11
pci3 at ppb2 bus 63
bge0 at pci3 dev 0 function 0 "Broadcom BCM5752" rev 0x01, BCM5752 A1 (0x6001): irq 10, address 00:15:60:9d:72:d9
brgphy0 at bge0 phy 1: BCM5752 10/100/1000baseT PHY, rev. 0
uhci0 at pci0 dev 29 function 0 "Intel 82801GB USB" rev 0x01: irq 5
uhci1 at pci0 dev 29 function 1 "Intel 82801GB USB" rev 0x01: irq 10
uhci2 at pci0 dev 29 function 2 "Intel 82801GB USB" rev 0x01: irq 11
uhci3 at pci0 dev 29 function 3 "Intel 82801GB USB" rev 0x01: irq 11
ehci0 at pci0 dev 29 function 7 "Intel 82801GB USB" rev 0x01: irq 5
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb3 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xe1
pci4 at ppb3 bus 5
ichpcib0 at pci0 dev 31 function 0 "Intel 82801GB LPC" rev 0x01: PM disabled
pciide0 at pci0 dev 31 function 1 "Intel 82801GB IDE" rev 0x01: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 disabled (no drives)
pciide1 at pci0 dev 31 function 2 "Intel 82801GB SATA" rev 0x01: DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide1: using irq 5 for native-PCI interrupt
wd0 at pciide1 channel 0 drive 0: <ST380819AS>
wd0: 16-sector PIO, LBA48, 76319MB, 156301488 sectors
wd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci3: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at ichpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
biomask fd6d netmask fd6d ttymask ffff
mtrr: Pentium Pro MTRR support
uhidev0 at uhub3 port 2 configuration 1 interface 0 "Logitech Optical USB Mouse" rev 2.00/3.40 addr 2
uhidev0: iclass 3/1
ums0 at uhidev0: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
softraid0 at root
root on wd0l swap on wd0b dump on wd0b

Le Fri, 11 Jul 2008 05:03:24 -0400,
Brad <brad@comstyle.com> a icrit :

> The following diff needs testing with PCIe bge(4) chipsets.
>
> To give people an idea of which chipsets are PCIe look for
> dmesg entries such as the following...
>
> bge0 at pci2 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1
> bge0 at pci2 dev 0 function 0 "Broadcom BCM5722" rev 0x00, BCM5755 C0
> bge0 at pci1 dev 0 function 0 "Broadcom BCM5751" rev 0x01, BCM5750 A1
> bge0 at pci2 dev 0 function 0 "Broadcom BCM5751M" rev 0x11, BCM5750 B1
> bge0 at pci4 dev 0 function 0 "Broadcom BCM5752" rev 0x02, BCM5752 A2
> bge0 at pci1 dev 0 function 0 "Broadcom BCM5753M" rev 0x21, BCM5750 C1
> bge0 at pci8 dev 0 function 0 "Broadcom BCM5754" rev 0x02,
> BCM5754/5787 A2 bge0 at pci5 dev 0 function 0 "Broadcom BCM5755M" rev
> 0x02, BCM5755 A2 bge0 at pci3 dev 0 function 0 "Broadcom BCM5787M"
> rev 0x02, BCM5754/5787 A2 bge0 at pci3 dev 0 function 0 "Broadcom
> BCM5789" rev 0x11, BCM5750 B1
>
> Please provide me with a dmesg.
>

[snip..]

Tested on my 2 computers (Optiplex GX520 and Latitude D630).

+ Optiplex
before:
bge0 at pci2 dev 0 function 0 "Broadcom BCM5751" rev 0x01, BCM5750 A1
(0x4001): apic 8 int 16 (irq 11), address 00:12:3f:97:1a:66
brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0

after:
bge0 at pci2 dev 0 function 0 "Broadcom BCM5751" rev 0x01, BCM5750 A1
(0x4001): irq 11, address 00:12:3f:97:1a:66
brgphy0 at bge0 phy 1:BCM5750 10/100/1000baseT PHY, rev. 0



+ Latitude:
before:
bge0 at pci3 dev 0 function 0 "Broadcom BCM5755M" rev 0x02, BCM5755 A2
(0xa002): irq 10, address 00:1c:23:07:40:3d
brgphy0 at bge0 phy 1: BCM5755 10/100/1000baseT PHY, rev. 0

after:
bge0 at pci3 dev 0 function 0 "Broadcom BCM5755M" rev 0x02, BCM5755 A2
(0xa002): irq 10, address 00:1c:23:07:40:3d
brgphy0 at bge0 phy 1: BCM5755 10/100/1000baseT PHY, rev. 0


Regards,


--
Pierre-Emmanuel Andre <pea at raveland.org>
GPG key: 0x7AE329DC

On Friday 11 July 2008 07:51:53 Pierre-Emmanuel Andri wrote:
> Le Fri, 11 Jul 2008 05:03:24 -0400,
>
> Brad <brad@comstyle.com> a icrit :
> > The following diff needs testing with PCIe bge(4) chipsets.
> >
> > To give people an idea of which chipsets are PCIe look for
> > dmesg entries such as the following...
> >
> > bge0 at pci2 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1
> > bge0 at pci2 dev 0 function 0 "Broadcom BCM5722" rev 0x00, BCM5755 C0
> > bge0 at pci1 dev 0 function 0 "Broadcom BCM5751" rev 0x01, BCM5750 A1
> > bge0 at pci2 dev 0 function 0 "Broadcom BCM5751M" rev 0x11, BCM5750 B1
> > bge0 at pci4 dev 0 function 0 "Broadcom BCM5752" rev 0x02, BCM5752 A2
> > bge0 at pci1 dev 0 function 0 "Broadcom BCM5753M" rev 0x21, BCM5750 C1
> > bge0 at pci8 dev 0 function 0 "Broadcom BCM5754" rev 0x02,
> > BCM5754/5787 A2 bge0 at pci5 dev 0 function 0 "Broadcom BCM5755M" rev
> > 0x02, BCM5755 A2 bge0 at pci3 dev 0 function 0 "Broadcom BCM5787M"
> > rev 0x02, BCM5754/5787 A2 bge0 at pci3 dev 0 function 0 "Broadcom
> > BCM5789" rev 0x11, BCM5750 B1
> >
> > Please provide me with a dmesg.
>
> [snip..]
>
> Tested on my 2 computers (Optiplex GX520 and Latitude D630).

Thanks. You will not see any change with the dmesg output. Just run
with things as usual and make sure there is no regression with regular
operation. Optionally, if you can do any performane benchmarking, as
I had asked Reyk to, this is expected to significantly improve transmit
performance.

> + Optiplex
> before:
> bge0 at pci2 dev 0 function 0 "Broadcom BCM5751" rev 0x01, BCM5750 A1
> (0x4001): apic 8 int 16 (irq 11), address 00:12:3f:97:1a:66
> brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
>
> after:
> bge0 at pci2 dev 0 function 0 "Broadcom BCM5751" rev 0x01, BCM5750 A1
> (0x4001): irq 11, address 00:12:3f:97:1a:66
> brgphy0 at bge0 phy 1:BCM5750 10/100/1000baseT PHY, rev. 0
>
>
>
> + Latitude:
> before:
> bge0 at pci3 dev 0 function 0 "Broadcom BCM5755M" rev 0x02, BCM5755 A2
> (0xa002): irq 10, address 00:1c:23:07:40:3d
> brgphy0 at bge0 phy 1: BCM5755 10/100/1000baseT PHY, rev. 0
>
> after:
> bge0 at pci3 dev 0 function 0 "Broadcom BCM5755M" rev 0x02, BCM5755 A2
> (0xa002): irq 10, address 00:1c:23:07:40:3d
> brgphy0 at bge0 phy 1: BCM5755 10/100/1000baseT PHY, rev. 0

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

On Friday 11 July 2008 08:01:01 Brad wrote:
> > Tested on my 2 computers (Optiplex GX520 and Latitude D630).
>
> Thanks. You will not see any change with the dmesg output. Just run
> with things as usual and make sure there is no regression with regular
> operation. Optionally, if you can do any performane benchmarking, as
> I had asked Reyk to, this is expected to significantly improve transmit
> performance.

Just to be clear since Pierre had sent me a benchmark in private. The
performance improvement would only be noticeable when running
on a Gigabit network. There should be no issue with saturating a
FastE connection as is.

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Le Fri, 11 Jul 2008 09:02:28 -0400,
Brad <brad@comstyle.com> a icrit :

> On Friday 11 July 2008 08:01:01 Brad wrote:
> > > Tested on my 2 computers (Optiplex GX520 and Latitude D630).
> >
> > Thanks. You will not see any change with the dmesg output. Just run
> > with things as usual and make sure there is no regression with
> > regular operation. Optionally, if you can do any performane
> > benchmarking, as I had asked Reyk to, this is expected to
> > significantly improve transmit performance.
>
> Just to be clear since Pierre had sent me a benchmark in private. The
> performance improvement would only be noticeable when running
> on a Gigabit network. There should be no issue with saturating a
> FastE connection as is.
>

I've forgotten to cc tech@.
Sorry.

--
Pierre-Emmanuel Andre <pea at raveland.org>
GPG key: 0x7AE329DC

Le Fri, 11 Jul 2008 09:02:28 -0400,
Brad <brad@comstyle.com> a icrit :

> On Friday 11 July 2008 08:01:01 Brad wrote:
> > > Tested on my 2 computers (Optiplex GX520 and Latitude D630).
> >
> > Thanks. You will not see any change with the dmesg output. Just run
> > with things as usual and make sure there is no regression with
> > regular operation. Optionally, if you can do any performane
> > benchmarking, as I had asked Reyk to, this is expected to
> > significantly improve transmit performance.
>
> Just to be clear since Pierre had sent me a benchmark in private. The
> performance improvement would only be noticeable when running
> on a Gigabit network. There should be no issue with saturating a
> FastE connection as is.
>

I've made an another test with my Latitude D630. Same thing with the
optiplex: performance improvement is not noticeable.
But there are no regressions.

Regards,

--
Pierre-Emmanuel Andre <pea at raveland.org>
GPG key: 0x7AE329DC

On Fri, Jul 11, 2008 at 03:44:48PM +0200, Pierre-Emmanuel Andr? wrote:
> I've made an another test with my Latitude D630. Same thing with the
> optiplex: performance improvement is not noticeable.
> But there are no regressions.
>

did you test it in both directions? the performance will only affect
the TX speed.

> Regards,
>
> --
> Pierre-Emmanuel Andre <pea at raveland.org>
> GPG key: 0x7AE329DC

Le Fri, 11 Jul 2008 16:26:49 +0200,
Reyk Floeter <reyk@openbsd.org> a icrit :

> On Fri, Jul 11, 2008 at 03:44:48PM +0200, Pierre-Emmanuel Andr? wrote:
> > I've made an another test with my Latitude D630. Same thing with the
> > optiplex: performance improvement is not noticeable.
> > But there are no regressions.
> >
>
> did you test it in both directions? the performance will only affect
> the TX speed.
>

I've made the same test as you (with iperf).


--
Pierre-Emmanuel Andre <pea at raveland.org>
GPG key: 0x7AE329DC