Peter wrote:
> Here's an update, thanks very much for the responses. I narrowed down the
> issue to: speed is only a problem in one direction. From one machine it's
> fast. The other direction is slow. The cards and machines are identical
> with the exception of 12GB RAM (ciddb) for one and 4GB RAM for the other
> (cid5). I did run netperf and TCP stream test:
>
> Direction Program Speed
> ----------------------------------------------------
> cid5 --> ciddb ftpget (2090.83 Kbytes/s)
> ciddb --> cid5 ftpget (32751.85 Kbytes/s)
>

Your problem is a difference in the limits of the filesystems on
the 2 machines, not anything to do with the network. ciddb's filesystem
is slower when writing. If you run a filesystem test, you'll see that.

Your TCP performance is exactly the same in both directions:

> # ./netperf -H ciddb
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to ciddb
> (172.17.1.109) port 0 AF_INET
> Recv Send Send
> Socket Socket Message Elapsed
> Size Size Size Time Throughput
> bytes bytes bytes secs. 10^6bits/sec
>
> 32768 32768 32768 10.02 94.69

> # ./netperf -H cid5
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to cid5 (172.17.1.111)
> port 0 AF_INET
> Recv Send Send
> Socket Socket Message Elapsed
> Size Size Size Time Throughput
> bytes bytes bytes secs. 10^6bits/sec
>
> 32768 32768 32768 10.01 94.80

As is the speed for *fiber, whatever that is:

> # ./netperf -H ciddbfiber
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to ciddbfiber
> (89.0.0.1) port 0 AF_INET
> Recv Send Send
> Socket Socket Message Elapsed
> Size Size Size Time Throughput
> bytes bytes bytes secs. 10^6bits/sec
>
> 32768 32768 32768 10.01 824.91

> # ./netperf -H cid5fiber
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to cid5fiber
> (89.0.0.2) port 0 AF_INET
> Recv Send Send
> Socket Socket Message Elapsed
> Size Size Size Time Throughput
> bytes bytes bytes secs. 10^6bits/sec
>
> 32768 32768 32768 10.01 821.51

As is the loopback performance:

> # ./netperf -H localhost
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost
> (127.0.0.1) port 0 AF_INET
> Recv Send Send
> Socket Socket Message Elapsed
> Size Size Size Time Throughput
> bytes bytes bytes secs. 10^6bits/sec
>
> 32768 32768 32768 10.02 4936.21
>
> # hostname
> ciddb
> # ./netperf -H localhost
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost
> (127.0.0.1) port 0 AF_INET
> Recv Send Send
> Socket Socket Message Elapsed
> Size Size Size Time Throughput
> bytes bytes bytes secs. 10^6bits/sec
>
> 32768 32768 32768 10.00 4991.00
>