SEO

New VMware workstation 5.5 has the ability to emulate 64-bit machines.
I took a shoot and tried it with new 3.8-amd64 release. At the first
glance, everything seemed to work fine. However, after the initial
reboot, I noticed a rather sluggish performance. Doing a simple 'make'
for a given port, the system crawls to it's knees.

After some debugging, I noticed a rather high/unusual number of
software interrupts. With a completely idle system, the number of
software interrupts is steady around 100. Another amd64 (native)
machine I have, it's around 2-3 when the system is idle.

Another thing I noticed:

pci1 at ppb0 bus 1 "Intel 82371AB PIIX4 ISA" rev 0x08 at pci0 dev 7
function 0 not configured

If I install 3.8-i386 on the same emulated 64-bit hardware (dmesg.i386
below) everything works like a charm.

Any pointers?

Output of vmstat (-i, -s and -m) is below, including dmesg.amd64 output.

Here are some stats:

# uptime
6:39AM up 21 mins, 1 user, load averages: 0.17, 0.24, 0.30

# vmstat -s | grep software
138409 software interrupts

138409 / (21 minutes * 60) = 110 si/sec

# vmstat -i
interrupt total rate
irq14/pciide0 13452 10
irq11/em0 12159 9
irq1/pckbc0 6 0
irq0/clock 124241 100
irq8/rtc 159028 128
Total 308886 248

# vmstat -s
4096 bytes per page
61357 pages managed
48050 pages free
1884 pages active
1291 pages inactive
0 pages being paged out
111 pages wired
0 pages zeroed
4 pages reserved for pagedaemon
6 pages reserved for kernel
65519 swap pages
0 swap pages in use
122759 total anon's in system
121665 free anon's
340272 page faults
315319 traps
295373 interrupts
39422 cpu context switches
324 fpu context switches
138409 software interrupts
463363 syscalls
0 pagein operations
0 swap ins
0 swap outs
2305 forks
7 forks where vmspace is shared
0 number of times the pagedaemon woke up
0 revolutions of the clock hand
0 pages freed by pagedaemon
0 pages scanned by pagedaemon
0 pages reactivated by pagedaemon
0 busy pages found by pagedaemon
50061 total name lookups
cache hits (81% pos + 4% neg) system 0% per-directory
deletions 0%, falsehits 0%, toolong 0%
0 select collisions

# vmstat -m

Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 1088 1216 233913 1280 14
32 639 641 131247 640 43
64 201 1143 20739 320 759
128 630 74 11804 160 4
256 94 34 2666 80 0
512 169 15 1994 40 0
1024 507 9 12068 20 0
2048 27 9 617 10 0
4096 25 2 2211 5 0
8192 3 0 3 5 0
16384 8 0 8 5 0
32768 1 0 1 5 0
65536 1 0 1 5 0

Memory usage type by bucket size
Size Type(s)
16 devbuf, pcb, sysctl, UFS mount, dirhash, exec, xform_data, VM swap,
UVM amap, UVM aobj, temp
32 devbuf, pcb, routetbl, ifaddr, namecache, sem, dirhash, in_multi,
exec, xform_data, UVM amap, temp
64 devbuf, pcb, routetbl, ifaddr, vnodes, dirhash, proc, VFS cluster,
in_multi, ether_multi, exec, VM swap, UVM amap, NDP, temp
128 devbuf, pcb, routetbl, ifaddr, sem, dirhash, ttys, pfkey data,
UVM amap, NDP
256 devbuf, routetbl, ifaddr, sysctl, vnodes, shm, UVM amap
512 devbuf, ifaddr, ioctlops, vnodes, VM map, dirhash, proc, NFS srvsock,
NFS daemon, ttys, newblk, UVM amap, temp
1024 devbuf, pcb, ioctlops, mount, UFS mount, shm, ttys, exec, UVM amap,
crypto data, temp
2048 devbuf, ifaddr, ioctlops, namecache, UFS mount, proc, VM swap,
UVM amap, UVM aobj
4096 devbuf, UFS mount, pagedep, UVM amap, temp
8192 devbuf, MSDOSFS mount, VM swap
16384 devbuf, NFS node, namecache, UFS quota, UFS mount, ISOFS mount,
inodedep, VM swap
32768 devbuf
65536 namecache

Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
devbuf 797 454K 456K 39099K 8953 0 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,327 68
pcb 16 5K 6K 39099K 65 0 0
16,32,64,128,1024
routetbl 63 7K 7K 39099K 85 0 0 32,64,128,256
ifaddr 43 13K 13K 39099K 43 0 0
32,64,128,256,512,2048
sysctl 2 1K 1K 39099K 2 0 0 16,256
ioctlops 0 0K 2K 39099K 24 0 0 512,1024,2048
mount 3 3K 4K 39099K 5 0 0 1024
NFS node 1 16K 16K 39099K 1 0 0 16384
vnodes 49 14K 88K 39099K 1267 0 0 64,256,512
namecache 4 83K 83K 39099K 4 0 0
32,2048,16384,65536
UFS quota 1 16K 16K 39099K 1 0 0 16384
UFS mount 13 38K 38K 39099K 13 0 0
16,1024,2048,4096,16384
shm 2 2K 2K 39099K 2 0 0 256,1024
VM map 3 2K 2K 39099K 3 0 0 512
sem 2 1K 1K 39099K 2 0 0 32,128
dirhash 24 5K 7K 39099K 69 0 0 16,32,64,128,512
proc 7 5K 5K 39099K 7 0 0 64,512,2048
VFS cluster 0 0K 1K 39099K 509 0 0 64
NFS srvsock 2 1K 1K 39099K 2 0 0 512
NFS daemon 1 1K 1K 39099K 1 0 0 512
in_multi 22 1K 1K 39099K 22 0 0 32,64
ether_multi 4 1K 1K 39099K 4 0 0 64
ISOFS mount 1 16K 16K 39099K 1 0 0 16384
MSDOSFS mount 1 8K 8K 39099K 1 0 0 8192
ttys 420 263K 263K 39099K 420 0 0 128,512,1024
exec 0 0K 3K 39099K 1743 0 0 16,32,64,1024
pfkey data 1 1K 1K 39099K 2 0 0 128
xform_data 0 0K 1K 39099K 83 0 0 16,32
pagedep 1 4K 4K 39099K 1 0 0 4096
inodedep 1 16K 16K 39099K 1 0 0 16384
newblk 1 1K 1K 39099K 1 0 0 512
VM swap 7 23K 23K 39099K 7 0 0
16,64,2048,8192,16384
UVM amap 1863 132K 220K 39099K 386101 0 0
16,32,64,128,256,512,1024,2048,4096
UVM aobj 2 3K 3K 39099K 2 0 0 16,2048
crypto data 1 1K 1K 39099K 1 0 0 1024
NDP 7 1K 1K 39099K 8 0 0 64,128
temp 34 5K 9K 39099K 17822 0 0
16,32,64,512,1024,4096

Memory Totals: In Use Free Requests
1130K 171K 417278
Memory resource pool statistics
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
extentpl 40 18 0 8 1 0 1 1 0 8 0
phpool 80 553 0 10 11 0 11 11 0 8 0
pcgpool 152 1 0 0 1 0 1 1 0 8 0
pmappl 224 2280 0 2265 2 0 2 2 0 8 1
pvpl 32 399987 0 395172 94 46 48 63 0 8 8
pdppl 4096 24 0 0 24 0 24 24 0 8 0
vmsppl 344 2280 0 2265 3 0 3 3 0 8 1
vmmpepl 144 216767 0 215810 207 165 42 66 0 8 6
vmmpekpl 144 5533 0 5529 1 0 1 1 0 8 0
aobjpl 88 1 0 0 1 0 1 1 0 8 0
amappl 56 127415 0 126795 18 0 18 18 0 8 8
bufpl 208 6 0 6 1 0 1 1 0 8 1
mbpl 256 22274 0 22013 29 10 19 19 1 8 2
mclpl 2048 8996 0 8740 138 0 138 138 4 3072 9
sockpl 360 112 0 89 3 0 3 3 0 8 0
procpl 544 2287 0 2265 5 0 5 5 0 8 1
zombiepl 144 2265 0 2265 1 0 1 1 0 8 1
ucredpl 80 59 0 51 1 0 1 1 0 8 0
pgrppl 40 46 0 32 1 0 1 1 0 8 0
sessionpl 64 20 0 7 1 0 1 1 0 8 0
pcredpl 32 2287 0 2265 1 0 1 1 0 8 0
lockfpl 88 6 0 4 1 0 1 1 0 8 0
filepl 80 10577 0 10519 2 0 2 2 0 8 0
fdescpl 424 2288 0 2265 4 0 4 4 0 8 1
pipepl 120 3266 0 3264 1 0 1 1 0 8 0
sigapl 472 2280 0 2265 3 0 3 3 0 8 1
wdcspl 176 13340 0 13340 1 0 1 1 0 8 1
scxspl 192 18 0 18 1 0 1 1 0 8 1
namei 1024 16637 0 16637 1 0 1 1 0 8 1
vnodes 272 2166 0 0 145 0 145 145 0 8 0
nchpl 112 1310 0 0 38 0 38 38 0 8 0
ffsino 336 4886 0 2724 181 0 181 181 0 8 0
dirhash 1024 78 0 49 10 0 10 10 0 8 2
rtentpl 208 29 0 0 2 0 2 2 0 8 0
tcpcbpl 552 22 0 9 3 0 3 3 0 8 1
tcpqepl 32 2 0 2 1 0 1 1 0 25 1
synpl 240 3 0 3 1 0 1 1 0 8 1
plimitpl 152 16 0 8 1 0 1 1 0 8 0
inpcbpl 328 56 0 40 2 0 2 2 0 8 0

In use 2564K, total allocated 2896K; utilization 88.5%


---- dmesg.amd64 ----

OpenBSD 3.8 (GENERIC) #247: Sat Sep 10 15:53:26 MDT 2005
deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 266924032 (260668K)
avail mem = 218468352 (213348K)
using 6567 buffers containing 26898432 bytes (26268K) of memory
mainbus0 (root)
cpu0 at mainbus0: (uniprocessor)
cpu0: AMD Athlon(tm) 64 Processor 3200+, 1994.76 MHz
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,P GE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE 3,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache
cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
pci0 at mainbus0 bus 0: configuration mode 1
pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01
pci1 at ppb0 bus 1
"Intel 82371AB PIIX4 ISA" rev 0x08 at pci0 dev 7 function 0 not configured
pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA,
channel 0 configured to compatibility, channel 1 configured to
compatibility
wd0 at pciide0 channel 0 drive 0: <VMware Virtual IDE Hard Drive>
wd0: 64-sector PIO, LBA, 4096MB, 8388608 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
atapiscsi0 at pciide0 channel 1 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <NECVMWar, VMware IDE CDR10, 1.00> SCSI0
5/cdrom removable
cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2
"Intel 82371AB Power" rev 0x08 at pci0 dev 7 function 3 not configured
vga1 at pci0 dev 15 function 0 "VMware Virtual SVGA II" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
mpt0 at pci0 dev 16 function 0 "Symbios Logic 53c1030" rev 0x01: irq 9
mpt0: IM support: 0
scsibus1 at mpt0: 16 targets
em0 at pci0 dev 17 function 0 "Intel PRO/1000MT (82545EM)" rev 0x01:
irq 11, address: 00:0c:29:02:db:f5
isa0 at mainbus0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
sysbeep0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
dkcsum: wd0 matches BIOS drive 0x80
root on wd0a
rootdev=0x0 rrootdev=0x300 rawdev=0x302

---- dmesg.i386 ----

OpenBSD 3.8 (GENERIC) #138: Sat Sep 10 15:41:37 MDT 2005
deraadt@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: AMD Athlon(tm) 64 Processor 3200+ ("AuthenticAMD" 686-class,
512KB L2 cache) 2 GHz
cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,P GE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE 3
cpu0: AMD Powernow: FID VID TTP TM STC
real mem = 267952128 (261672K)
avail mem = 237481984 (231916K)
using 3296 buffers containing 13500416 bytes (13184K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(ff) BIOS, date 07/29/05, BIOS32 rev. 0 @ 0xfd880
apm0 at bios0: Power Management spec V1.2
apm0: AC on, battery charge unknown
apm0: flags 30102 dobusy 0 doidle 1
pcibios0 at bios0: rev 2.1 @ 0xfd880/0x780
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfdf30/176 (9 entries)
pcibios0: PCI Interrupt Router at 000:07:0 ("Intel 82371FB ISA" rev 0x00)
pcibios0: PCI bus #1 is the last bus
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000 0xdc000/0x4000! 0xe0000/0x4000!
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01
pci1 at ppb0 bus 1
pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08
pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA,
channel 0 configured to compatibility, channel 1 configured to
compatibility
wd0 at pciide0 channel 0 drive 0: <VMware Virtual IDE Hard Drive>
wd0: 64-sector PIO, LBA, 4096MB, 8388608 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
atapiscsi0 at pciide0 channel 1 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <NECVMWar, VMware IDE CDR10, 1.00> SCSI0
5/cdrom removable
cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2
"Intel 82371AB Power" rev 0x08 at pci0 dev 7 function 3 not configured
vga1 at pci0 dev 15 function 0 "VMware Virtual SVGA II" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
mpt0 at pci0 dev 16 function 0 "Symbios Logic 53c1030" rev 0x01: irq 9
mpt0: IM support: 0
scsibus1 at mpt0: 16 targets
em0 at pci0 dev 17 function 0 "Intel PRO/1000MT (82545EM)" rev 0x01:
irq 11, address: 00:0c:29:02:db:f5
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
sysbeep0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
biomask e765 netmask ef65 ttymask ffe7
pctr: user-level cycle counter enabled
dkcsum: wd0 matches BIOS drive 0x80
root on wd0a
rootdev=0x0 rrootdev=0x300 rawdev=0x302