Rick Jones wrote:

> Peter <gowf67@yahoo.com> wrote:
>> Hello, I've got 2 Itanium Systems running (B.11.23 U ia64). Both
>> systems have the 1000Base-SX cards installed. The settings for both
>> are as follows:
>
> IIRC There are at least three possible 1000Base-SX cards - an old one
> based on Tigon2, a newer one based on Tigon3/BCM5701 and then a
> dual-port based on Intel - whcich specifically is it? (ioscand -fk
> may help there). (Although it may be that the dual-port is just
> copper)
>
> Also, which model of Itanium systems and at which frequency?
>
>> Speed = 1000
>> Duplex Mode = Full
>> Autoneg = On
>> MTU = 9000bytes
>> Flow Control = On
>> Send Max Buffers = 10
>> Receive Max Buffers = 1
>> Send Coal Ticks = 150
>> Receive Coal Ticks = 0
>
>> Now, the broadcast for both is 89.255.255.255 and subnet 255.0.0.0 IP
>> 89.0.0.1(2) respectively.
>
>> The two machines are connected back to back via fiber crossover. The
>> throughput for large files is horrible. Like 10BaseT or worse. For
>> small files <1GB , its fast. Which may be due to disk caching???? We
>> need the large file throughput to be maxed out. The drives are 160MB
>> per sec SCSI so I know we are not even close to that yet.
>
> Which protocol/application are you using to transfer the files? FTP?
> NFS? Samba? scp? Home-brew?
>
>> I wonder if my IP settings are too loose?
>
>> I'm at a loss now, Thanks in advance for your help.
>
> Step one - eliminate variables. Run a bandwidth test that does not
> involve a filesystem - eg netperf - http://www.netperf.org/ - and the
> TCP_STREAM test.
>
> Step one check for packet losses:
>
> netstat -p tcp > before;your test;netstat -p tcp > after
> beforeafter before after > delta
> examine delta
>
> (beforeafter from ftp://ftp.cup.hp.com/dist/networking/tools)
>
> Do something similar with lanadmin stats: lanadmin -g mibstats <ppa> ...
>
> and we can go from there.
>
> rick jones
Rick Jones wrote:

> Peter <gowf67@yahoo.com> wrote:
>> Hello, I've got 2 Itanium Systems running (B.11.23 U ia64). Both
>> systems have the 1000Base-SX cards installed. The settings for both
>> are as follows:
>
> IIRC There are at least three possible 1000Base-SX cards - an old one
> based on Tigon2, a newer one based on Tigon3/BCM5701 and then a
> dual-port based on Intel - whcich specifically is it? (ioscand -fk
> may help there). (Although it may be that the dual-port is just
> copper)
>
> Also, which model of Itanium systems and at which frequency?
>
>> Speed = 1000
>> Duplex Mode = Full
>> Autoneg = On
>> MTU = 9000bytes
>> Flow Control = On
>> Send Max Buffers = 10
>> Receive Max Buffers = 1
>> Send Coal Ticks = 150
>> Receive Coal Ticks = 0
>
>> Now, the broadcast for both is 89.255.255.255 and subnet 255.0.0.0 IP
>> 89.0.0.1(2) respectively.
>
>> The two machines are connected back to back via fiber crossover. The
>> throughput for large files is horrible. Like 10BaseT or worse. For
>> small files <1GB , its fast. Which may be due to disk caching???? We
>> need the large file throughput to be maxed out. The drives are 160MB
>> per sec SCSI so I know we are not even close to that yet.
>
> Which protocol/application are you using to transfer the files? FTP?
> NFS? Samba? scp? Home-brew?
>
>> I wonder if my IP settings are too loose?
>
>> I'm at a loss now, Thanks in advance for your help.
>
> Step one - eliminate variables. Run a bandwidth test that does not
> involve a filesystem - eg netperf - http://www.netperf.org/ - and the
> TCP_STREAM test.
>
> Step one check for packet losses:
>
> netstat -p tcp > before;your test;netstat -p tcp > after
> beforeafter before after > delta
> examine delta
>
> (beforeafter from ftp://ftp.cup.hp.com/dist/networking/tools)
>
> Do something similar with lanadmin stats: lanadmin -g mibstats <ppa> ...
>
> and we can go from there.
>
> rick jones


Here's an update, thanks very much for the responses. I narrowed down the
issue to: speed is only a problem in one direction. From one machine it's
fast. The other direction is slow. The cards and machines are identical
with the exception of 12GB RAM (ciddb) for one and 4GB RAM for the other
(cid5). I did run netperf and TCP stream test:

Direction Program Speed
----------------------------------------------------
cid5 --> ciddb ftpget (2090.83 Kbytes/s)
ciddb --> cid5 ftpget (32751.85 Kbytes/s)


TCP Stream Performance

# hostname
cid5
# ./netperf -H ciddbfiber
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to ciddbfiber
(89.0.0.1) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

32768 32768 32768 10.01 824.91
# ./netperf -H localhost
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost
(127.0.0.1) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

32768 32768 32768 10.02 4936.21
# ./netperf -H ciddb
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to ciddb
(172.17.1.109) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

32768 32768 32768 10.02 94.69

# hostname
ciddb
# ./netperf -H cid5fiber
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to cid5fiber
(89.0.0.2) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

32768 32768 32768 10.01 821.51
# ./netperf -H localhost
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost
(127.0.0.1) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

32768 32768 32768 10.00 4991.00
# ./netperf -H cid5
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to cid5 (172.17.1.111)
port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

32768 32768 32768 10.01 94.80

Here's the card model:
lan 1 0/7/2/0 igelan CLAIMED INTERFACE HP A6847-60101 PCI 1000Base-SX
Adapter

The other issue is the kernel parameters are not the same:

Here's the lovely parameters diff. Thanks again for your help and time.

Tunable Value Exp Tunable Value
Express
NSTREVENT 50 Def NSTREVENT 50
Default
NSTRPUSH 16 Def NSTRPUSH 16
Default
NSTRSCHED 0 Def NSTRSCHED 0
Default
STRCTLSZ 1024 Def STRCTLSZ 1024
Default
STRMSGSZ 65535 655 | STRMSGSZ 0
Default
acctresume 4 Def acctresume 4
Default
acctsuspend 2 Def acctsuspend 2
Default
aio_listio_max 256 Def aio_listio_max 256
Default
aio_max_ops 2048 Def aio_max_ops 2048
Default
aio_monitor_run_sec 30 Def aio_monitor_run_sec 30
Default
aio_physmem_pct 10 Def aio_physmem_pct 10
Default
aio_prio_delta_max 20 Def aio_prio_delta_max 20
Default
aio_proc_thread_pct 70 Def aio_proc_thread_pct 70
Default
aio_proc_threads 1024 Def aio_proc_threads 1024
Default
aio_req_per_thread 1 Def aio_req_per_thread 1
Default
allocate_fs_swapmap 0 Def allocate_fs_swapmap 0
Default
alwaysdump 1 1 | alwaysdump 0
Default
bufcache_hash_locks 128 Def bufcache_hash_locks 128
Default
chanq_hash_locks 256 Def chanq_hash_locks 256
Default
core_addshmem_read 0 Def core_addshmem_read 0
Default
core_addshmem_write 0 Def core_addshmem_write 0
Default
create_fastlinks 1 1 | create_fastlinks 0
Default
dbc_max_pct 8 8 | dbc_max_pct 50
Default
dbc_min_pct 5 Def dbc_min_pct 5
Default
default_disk_ir 0 Def default_disk_ir 0
Default
disksort_seconds 0 Def disksort_seconds 0
Default
dma32_pool_size 268435456 Def dma32_pool_size 268435456
Default
dmp_rootdev_is_vol 0 Def dmp_rootdev_is_vol 0
Default
dmp_swapdev_is_vol 0 Def dmp_swapdev_is_vol 0
Default
dnlc_hash_locks 512 Def dnlc_hash_locks 512
Default
dontdump 0 Def dontdump 0
Default
dst 1 Def dst 1
Default
dump_compress_on 1 Def dump_compress_on 1
Default
enable_idds 0 Def enable_idds 0
Default
eqmemsize 15 Def eqmemsize 15
Default
executable_stack 1 1 | executable_stack 0
Default
fs_async 1 1 | fs_async 0
Default
fs_symlinks 20 Def fs_symlinks 20
Default
ftable_hash_locks 64 Def ftable_hash_locks 64
Default
hp_hfs_mtra_enabled 1 Def hp_hfs_mtra_enabled 1
Default
io_ports_hash_locks 64 Def io_ports_hash_locks 64
Default
ksi_alloc_max 33600 Def | ksi_alloc_max 33600
(nproc*
ksi_send_max 32 Def ksi_send_max 32
Default
max_acct_file_size 2560000 Def max_acct_file_size 2560000
Default
max_async_ports 50 Def max_async_ports 50
Default
max_mem_window 0 Def max_mem_window 0
Default
max_thread_proc 3000 300 | max_thread_proc 256
Default
maxdsiz 0x12000000 0x1 | maxdsiz 1073741824
Default
maxdsiz_64bit 4294967296 Def maxdsiz_64bit 4294967296
Default
maxfiles 2048 204 | maxfiles 2048
Default
maxfiles_lim 2048 204 | maxfiles_lim 4096
Default
maxrsessiz 8388608 Def maxrsessiz 8388608
Default
maxrsessiz_64bit 8388608 Def maxrsessiz_64bit 8388608
Default
maxssiz 8388608 Def | maxssiz 134217728
1342177
maxssiz_64bit 268435456 Def maxssiz_64bit 268435456
Default
maxtsiz 100663296 Def maxtsiz 100663296
Default
maxtsiz_64bit 1073741824 Def maxtsiz_64bit 1073741824
Default
maxuprc 2048 204 | maxuprc 4195
(nproc-
maxvgs 30 30 | maxvgs 10
Default
msgmap 6642 664 | msgmap 4202
(nproc+
msgmax 65535 655 | msgmax 8192
Default
msgmnb 65535 655 | msgmnb 16384
Default
msgmni 6200 620 | msgmni 4200
nproc
msgseg 32767 327 | msgseg 8192
Default
msgssz 96 Def msgssz 96
Default
msgtql 6640 664 | msgtql 4200
nproc
ncdnode 150 Def ncdnode 150
Default
nclist 8292 Def nclist 8292
Default
ncsize 15680 156 | ncsize 8976
Default
nfile 4097 409 | nfile 65048
(15*npr
nflocks 4096 Def | nflocks 4200
nproc
ninode 7500 750 | ninode 35648
(8*npro
nkthread 6000 600 | nkthread 7366
(((npro
nproc 4096 409 | nproc 4200
Default
npty 512 512 | npty 60
Default
nstrpty 60 60 nstrpty 60
60
nstrtel 60 Def nstrtel 60
Default
nswapdev 10 Def nswapdev 10
Default
nswapfs 10 Def nswapfs 10
Default
nsysmap 8400 Def nsysmap 8400
Default
nsysmap64 8400 Def nsysmap64 8400
Default
o_sync_is_o_dsync 0 Def o_sync_is_o_dsync 0
Default
pa_maxssiz_32bit 83648512 Def pa_maxssiz_32bit 83648512
Default
pa_maxssiz_64bit 536870912 Def pa_maxssiz_64bit 536870912
Default
pfdat_hash_locks 128 Def pfdat_hash_locks 128
Default
physical_io_buffers 768 Def physical_io_buffers 768
Default
region_hash_locks 128 Def region_hash_locks 128
Default
remote_nfs_swap 0 Def remote_nfs_swap 0
Default
rng_bitvals 9876543210 Def rng_bitvals 9876543210
Default
rng_sleeptime 2 Def rng_sleeptime 2
Default
rtsched_numpri 32 Def rtsched_numpri 32
Default
scroll_lines 100 Def scroll_lines 100
Default
scsi_max_qdepth 8 Def scsi_max_qdepth 8
Default
scsi_maxphys 1048576 Def scsi_maxphys 1048576
Default
secure_sid_scripts 1 Def secure_sid_scripts 1
Default
semaem 16384 Def semaem 16384
Default
semmni 2400 240 | semmni 8400
(nproc*
semmns 16384 163 | semmns 16800
(nproc*
semmnu 600 600 | semmnu 4196
(nproc-
semmsl 2048 Def semmsl 2048
Default
semume 256 256 | semume 100
Default
semvmx 32767 Def semvmx 32767
Default
sendfile_max 0 Def sendfile_max 0
Default
shmmax 0x40000000 0x4 | shmmax 16737418240
1673741
shmmni 512 512 | shmmni 400
Default
shmseg 300 Def shmseg 300
Default
st_ats_enabled 0 Def st_ats_enabled 0
Default
st_fail_overruns 0 Def st_fail_overruns 0
Default
st_large_recs 0 Def st_large_recs 0
Default
streampipes 0 Def streampipes 0
Default
swapmem_on 1 Def swapmem_on 1
Default
swchunk 2048 Def swchunk 2048
Default
sysv_hash_locks 128 Def sysv_hash_locks 128
Default
tcphashsz 2048 Def tcphashsz 2048
Default
timeslice 10 Def timeslice 10
Default
timezone 420 Def timezone 420
Default
unlockable_mem 0 Def unlockable_mem 0
Default
vnode_cd_hash_locks 128 Def vnode_cd_hash_locks 128
Default
vnode_hash_locks 128 Def vnode_hash_locks 128
Default
vol_checkpt_default 10240 Def vol_checkpt_default 10240
Default
vol_dcm_replay_size 262144 Def vol_dcm_replay_size 262144
Default
vol_default_iodelay 50 Def vol_default_iodelay 50
Default
vol_fmr_logsz 4 Def vol_fmr_logsz 4
Default
vol_max_bchain 32 Def vol_max_bchain 32
Default
vol_max_nconfigs 20 Def vol_max_nconfigs 20
Default
vol_max_nlogs 20 Def vol_max_nlogs 20
Default
vol_max_nmpool_sz 4194304 Def vol_max_nmpool_sz 4194304
Default
vol_max_prm_dgs 1024 Def vol_max_prm_dgs 1024
Default
vol_max_rdback_sz 4194304 Def vol_max_rdback_sz 4194304
Default
vol_max_vol 8388608 Def vol_max_vol 8388608
Default
vol_maxio 256 Def vol_maxio 256
Default
vol_maxioctl 32768 Def vol_maxioctl 32768
Default
vol_maxkiocount 2048 Def vol_maxkiocount 2048
Default
vol_maxparallelio 256 Def vol_maxparallelio 256
Default
vol_maxspecialio 256 Def vol_maxspecialio 256
Default
vol_maxstablebufsize 256 Def vol_maxstablebufsize 256
Default
vol_min_lowmem_sz 524288 Def vol_min_lowmem_sz 524288
Default
vol_mvr_maxround 256 Def vol_mvr_maxround 256
Default
vol_nm_hb_timeout 10 Def vol_nm_hb_timeout 10
Default
vol_rootdev_is_vol 0 Def vol_rootdev_is_vol 0
Default
vol_subdisk_num 4096 Def vol_subdisk_num 4096
Default
vol_swapdev_is_vol 0 Def vol_swapdev_is_vol 0
Default
vol_vvr_transport 1 Def vol_vvr_transport 1
Default
vol_vvr_use_nat 0 Def vol_vvr_use_nat 0
Default
volcvm_cluster_size 4 Def volcvm_cluster_size 4
Default
volcvm_smartsync 1 Def volcvm_smartsync 1
Default
voldrl_max_drtregs 2048 Def voldrl_max_drtregs 2048
Default
voldrl_min_regionsz 512 Def voldrl_min_regionsz 512
Default
voliomem_chunk_size 65536 Def voliomem_chunk_size 65536
Default
voliomem_maxpool_sz 4194304 Def voliomem_maxpool_sz 4194304
Default
voliot_errbuf_dflt 16384 Def voliot_errbuf_dflt 16384
Default
voliot_iobuf_default 8192 Def voliot_iobuf_default 8192
Default
voliot_iobuf_limit 131072 Def voliot_iobuf_limit 131072
Default
voliot_iobuf_max 65536 Def voliot_iobuf_max 65536
Default
voliot_max_open 32 Def voliot_max_open 32
Default
volraid_rsrtransmax 1 Def volraid_rsrtransmax 1
Default
vps_ceiling 16 Def vps_ceiling 16
Default
vps_chatr_ceiling 1048576 Def vps_chatr_ceiling 1048576
Default
vps_pagesize 4 Def vps_pagesize 4
Default
vx_maxlink 32767 Def vx_maxlink 32767
Default
vx_ninode 0 Def vx_ninode 0
Default
vxfs_bc_bufhwm 0 Def vxfs_bc_bufhwm 0
Default
vxfs_ifree_timelag 0 Def vxfs_ifree_timelag 0
Default
vxtask_max_monitors 32 Def vxtask_max_monitors 32
Default