Re: Informix Performance

#1 (**permalink**) 04-19-2008, 09:34 PM

--0-770701732-1077895028=:2607
Content-Type: text/plain; charset=us-ascii

Daniel,

Here are the answers by the topic number that u are having problems with.

1. The raw devices is the best because the IO is faster on raw devices. Have your sysadmin create rawdevices for you. (Informix 9.4 allows chunks to be created larger than two gigs, so please upgrade to 9.4 to take advantage of large chunks ).
2. Ok. onstat -g seg shows that your IDS is dynamically allocating virtual segments !! This not good. Increase your SHMVRTSIZE.
4. Ok, your onstat -u shows that the processes are not waiting for buffers, which is a good thing.
10. Ok, HERE IS YOUR PROBLEM ! Your IDS is stuck at the check point, this means you are trying to write a whole bunch of stuff to disk at the check point and when this happens IDS has to slow down because it has to write all the data into the disk. Now the solution for this is to increase your LRU writes. In order to do this you must know how to tune LRU_MAX_DIRTY and LRU_MIN_DIRTY parameters. Before you mess with those parameters please read about them and test them in the test box and get a handle of how it works. I can guarantee you that your checkpoint duration is very long. You can check that on your online.log. The 2 most important commands to figure out how to tune these are onstat -F and onstat -R. The outputs from them will give u an idea how your LRUs and Cleaners are behaving. Also, you are doing your check point in every 10 minutes, try to do it every 5 minutes or less and test to see how the engine reacts. When u wait for a whole 10 minutes the IDS may have
gathered a whole bunch of stuff to write to disk.

10. Try not to use the default. Edit your onconfig and enter NETTYPE on it.
13. This will give u more cleaners to do the work. Cleaners do their work and go to sleep. More Cleaners u have is the better but don't go haywire on that. Increase is gradually and monitor !
14. sar 2 20 will execute sar in every 2 seconds for 20 times. If the %wio column of the output of this command shows you high numbers (50,80,90) that means you have to tune your OS, your OS is waiting for blocks, I don't know much sysadmin stuff, but know enough to say thats a big problem.
17. Ok, ipcs is important ! This will show you what memory segments are allocated to informix from the OS. Now, you need to learn how to map these memory segments to Informix onstat -g seg output, another words how to map them to your Informix server. Now, you have to be very carefull, if you are running more than one Informix instance on this machine, then you need to map them to all those instances and figure out which ones are useless and get rid of the ones that are useless.
18. Yes, this is another area you should only mess with after reading about it and knowing how they work. Your /etc/system will show u the configuration. Increase the sem* parameters in this file gradually and test in the test machine.

I know this is too much. Everyone, please chip in areas I may have explained vaguely.

Hope this helps.

Ravi Thero.
Daniel Castro <daniel.castro@feliciorocho.org.br> wrote:
Ravi T wrote:

> You seem to have many misconfigurations on your Informix server !!!
>
> Start investigating the following areas !
>
> 1. Why are u using cooked files ?
> 2. How does your onstat -g seg looks like when the slow down happens ?
> 3. Check all the processes that are running when the slow down
> happens. Backups, update statistics, batch jobs etc etc.
> 4. How does the onstat -u look like when the slow down happens. Are
> there processes waiting for buffers ?
> 5. What kind of locking are you using ? Page level locks, row locks etc.
> 6. When users complain of slowness you need to see what they are
> trying to do and check if the tables they are trying to hit are locked
> at the time.
> 7. When the slowness happens are there logical logs that have not
> been backep up ?
> 8. Did you change any /etc/system parameters ? If so did u reboot
> the Unix server after doing so ?
> 9. How does your onstat -g ntt look like ? Does it show processes
> that are very old and just sitting there ?
> 10. What kind of connections are you using and how many ? Is your
> Nettype paramer configured correctly ? By the way you don't even show
> NETTYPE parameter on your onconfig file that you have pasted here.
> 11. Have you tried the add extra CPUVPS dynamically when slowness
> happens.
> 12. Is your TEMP dpspace big enough ? Can you add more TEMP
> spaces and monitor the system. Its better to have several TEMP spaces
> rather than having one big TEMP space.
> 13. Increase the CLEANERS to 5 or 6 and monitor the system.
> 14. From the OS side when you issue sar command does it show that OS
> is waiting for block writes ?
> 15. Check onstat -g dic, mem, stk, ses to figure out the memory and
> caching by process.
> 16. Reset your profile counts using onstat -z and monitor onstat -p
> 17. Use ipcs to lookat memory from OS side and see if there is memory
> allocated other stuff other than Informix and investigate.
> 18. See if u are allocating enough semophores at the IDS start.
>
> Those abvoe thoughts are some things to check to start with.
>
> Ravi Thero
>
> */Daniel Castro /* wrote:
>
> Hi,
>
> I am having informix performance problems. I don´t know where to
> start.
> Sometimes the database hangs, always from 00:00am to 5:00 am, but it
> gives no error in the log file. Can you help me?
> Sometimes the users says that the system is slow, but I can´t identify
> why.
> First, the result of onstat -p:
>
> Informix Dynamic Server Version 9.30.UC1 -- On-Line -- Up 9 days
> 21:06:09 -- 1582656 Kbytes
>
> Profile
> dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
> 67223627 173897813 2077681908 96.76 12042759 61104633 222993237
> 94.60
>
> isamtot open start read write rewrite delete commit
> rollbk
> 2790216351 184683421 616967012 4148277112 65017319 12219659 28562330
> 8807460 884
>
> gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
> 0 0 0 0 0 0 0
>
> ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
> 7 0 0 336038.88 116702.45 1385 4099
>
> bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress
> seqscans
> 7192942 14028 903998567 4 0 6224 6824574
> 39496093
>
> ixda-RA idx-RA da-RA RA-pgsused lchwaits
> 12311760 5075181 15389056 32074507 56519934
>
> Second my onconfig file:
>
> #************************************************* *************************
> #
> # INFORMIX SOFTWARE, INC.
> #
> # Title: onconfig.producao
> # Description: Informix Dynamic Server Configuration Parameters
> #
> #************************************************* *************************
> # Root Dbspace Configuration
> ROOTNAME rootdbs # Root dbspace name
> ROOTPATH /cooked_file/dbs_rootP # Path for device containing
> root dbspace
> ROOTOFFSET 0 # Offset of root dbspace into device
> (Kbytes)
> ROOTSIZE 471039 # Size of root dbspace (Kbytes)
> # Disk Mirroring Configuration Parameters
> MIRROR 0 # Mirroring flag (Yes = 1, No = 0)
> MIRRORPATH # Path for device containing mirrored
> root
> MIRROROFFSET 0 # Offset into mirrored device (Kbytes)
> # Physical Log Configuration
> PHYSDBS wpdphlog # Location (dbspace) of physical log
> PHYSFILE 60000 # Physical log file size (Kbytes)
> # Logical Log Configuration
> LOGFILES 40 # Number of logical log files
> LOGSIZE 10000 # Logical log size (Kbytes)
> # Diagnostics
> MSGPATH /opt/informix/producao.log # System message log file
> path
> CONSOLE /dev/console # System console message path
> ALARMPROGRAM /opt/informix/etc/log_full.sh # Alarm program path
> TBLSPACE_STATS 1 # Maintain tblspace statistics
> # System Archive Tape Device
> TAPEDEV /dev/st0 # Tape device path
> TAPEBLK 16 # Tape block size (Kbytes)
> TAPESIZE 100000000 # Maximum amount of data to put on
> tape (Kbytes)
> # Log Archive Tape Device
> LTAPEDEV /dev/st0 # Log tape device path
> LTAPEBLK 16 # Log tape block size (Kbytes)
> LTAPESIZE 100000000 # Max amount of data to put on log
> tape (Kbytes)
> # Optical
> STAGEBLOB # Informix Dynamic Server staging area
> # System Configuration
> SERVERNUM 1 # Unique id corresponding to a OnLine
> instance
> DBSERVERNAME producao # Name of default database server
> DBSERVERALIASES # List of alternate dbservernames
> DEADLOCK_TIMEOUT 60 # Max time to wait of lock in
> distributed env.
> RESIDENT 0 # Forced residency flag (Yes = 1, No =
> 0)
> MULTIPROCESSOR 1 # 0 for single-processor, 1 for
> multi-processor
> # NUMCPUVPS 1 # Number of user (cpu) vps
> SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps
> to one
> # NOAGE 0 # Process aging
> # AFF_SPROC 0 # Affinity start processor
> # AFF_NPROCS 0 # Affinity number of processors
> # Shared Memory Parameters
> LOCKS 200000 # Maximum number of locks
> BUFFERS 600000 # Maximum number of shared buffers
> # NUMAIOVPS # Number of IO vps
> PHYSBUFF 128 # Physical log buffer size (Kbytes)
> LOGBUFF 128 # Logical log buffer size (Kbytes)
> CLEANERS 4 # Number of buffer cleaner processes
> SHMBASE 0x50000000 # Shared memory base address
> SHMVIRTSIZE 154000 # initial virtual shared memory
> segment size
> SHMADD 32768 # Size of new shared memory segments
> (Kbytes)
> SHMTOTAL 0 # Total shared memory (Kbytes).
> 0=>unlimited
> CKPTINTVL 600 # Check point interval (in sec)
> LRUS 64 # Number of LRU queues
> LRU_MAX_DIRTY 3 # LRU percent dirty begin cleaning
> limit
> LRU_MIN_DIRTY 1 # LRU percent dirty end cleaning limit
> TXTIMEOUT 0x12c # Transaction timeout (in sec)
> STACKSIZE 32 # Stack size (Kbytes)
> # System Page Size
> # BUFFSIZE - OnLine no longer supports this configuration parameter.
> # To determine the page size used by OnLine on your
> platform
> # see the last line of output from the command, 'onstat
> -b'.
> # Recovery Variables
> # OFF_RECVRY_THREADS:
> # Number of parallel worker threads during fast recovery or an offline
> restore.
> # ON_RECVRY_THREADS:
> # Number of parallel worker threads during an online restore.
> OFF_RECVRY_THREADS 10 # Default number of offline worker
> threads
> ON_RECVRY_THREADS 1 # Default number of online worker
> threads
> # Data Replication Variables
> DRINTERVAL 30 # DR max time between DR buffer
> flushes (in sec)
> DRTIMEOUT 30 # DR network timeout (in sec)
> DRLOSTFOUND /usr/informix/etc/dr.lostfound # DR lost+found file
> path
> # CDR Variables
> CDR_EVALTHREADS 1,2 # evaluator threads
> (per-cpu-vp,additional)
> CDR_DSLOCKWAIT 5 # DS lockwait timeout (seconds)
> CDR_QUEUEMEM 4096 # Maximum amount of memory for any CDR
> queue (Kbytes)
> CDR_NIFCOMPRESS 0 # Link level compression (-1 never, 0
> none, 9 max)
> CDR_SERIAL 0,0 # Serial Column Sequence
> CDR_DBSPACE # dbspace for syscdr database
> CDR_QHDR_DBSPACE # CDR queue dbspace (default same as
> catalog)
> CDR_QDATA_SBSPACE # CDR queue smart blob space
> CDR_QDATA_SBFLAGS 0 # Log/no-log (default no log)
> # Backup/Restore variables
> BAR_ACT_LOG /home/informix/bar_act.log
> # ON-Bar Log file - not in /tmp please
> BAR_DEBUG_LOG /home/informix/bar_dbug.log
> # ON-Bar Debug Log - not in /tmp
> please
> BAR_MAX_BACKUP 0
> BAR_RETRY 1
> BAR_NB_XPORT_COUNT 10
> BAR_XFER_BUF_SIZE 31
> RESTARTABLE_RESTORE on
> BAR_PROGRESS_FREQ 0
> # Informix Storage Manager variables
> ISM_DATA_POOL ISMData
> ISM_LOG_POOL ISMLogs
> # Read Ahead Variables
> RA_PAGES 32 # Number of pages to attempt to read
> ahead
> RA_THRESHOLD 20 # Number of pages left before next
> group
> # DBSPACETEMP:
> # OnLine equivalent of DBTEMP for SE. This is the list of dbspaces
> # that the OnLine SQL Engine will use to create temp tables etc.
> # If specified it must be a colon separated list of dbspaces that
> exist
> # when the OnLine system is brought online. If not specified, or if
> # all dbspaces specified are invalid, various ad hoc queries will
> create
> # temporary files in /tmp instead.
> DBSPACETEMP wpdtemp1 # Default temp dbspaces
> # DUMP*:
> # The following parameters control the type of diagnostics information
> which
> # is preserved when an unanticipated error condition (assertion
> failure) occurs
> # during OnLine operations.
> # For DUMPSHMEM, DUMPGCORE and DUMPCORE 1 means Yes, 0 means No.
> DUMPDIR /tmp # Preserve diagnostics in this
> directory
> DUMPSHMEM 1 # Dump a copy of shared memory
> DUMPGCORE 0 # Dump a core image using 'gcore'
> DUMPCORE 0 # Dump a core image (Warning:this
> aborts OnLine)
> DUMPCNT 1 # Number of shared memory or gcore
> dumps for
> # a single user's session
> FILLFACTOR 90 # Fill factor for building indexes
> # method for OnLine to use when determining current time
> USEOSTIME 0 # 0: use internal time(fast), 1: get
> time from OS(slow)
> # Parallel Database Queries (pdq)
> MAX_PDQPRIORITY 100 # Maximum allowed pdqpriority
> DS_MAX_QUERIES # Maximum number of decision support
> queries
> DS_TOTAL_MEMORY # Decision support memory (Kbytes)
> DS_MAX_SCANS 1048576 # Maximum number of decision support
> scans
> DATASKIP off # List of dbspaces to skip
> # OPTCOMPIND
> # 0 => Nested loop joins will be preferred (where
> # possible) over sortmerge joins and hash joins.
> # 1 => If the transaction isolation mode is not
> # "repeatable read", optimizer behaves as in (2)
> # below. Otherwise it behaves as in (0) above.
> # 2 => Use costs regardless of the transaction isolation
> # mode. Nested loop joins are not necessarily
> # preferred. Optimizer bases its decision purely
> # on costs.
> OPTCOMPIND 2 # To hint the optimizer
> DIRECTIVES 1 # Optimizer DIRECTIVES ON (1/Default)
> or OFF (0)
> ONDBSPACEDOWN 2 # Dbspace down option: 0 = CONTINUE, 1
> = ABORT, 2 = WAIT
> OPCACHEMAX 0 # Maximum optical cache size (Kbytes)
> # HETERO_COMMIT (Gateway participation in distributed transactions)
> # 1 => Heterogeneous Commit is enabled
> # 0 (or any other value) => Heterogeneous Commit is disabled
> HETERO_COMMIT 0
> SBSPACENAME # Default smartblob space name - this
> is where blobs
> # go if no sbspace is specified when the smartblob is
> # created. It is also used by some datablades as
> # the location to put their smartblobs.
> SYSSBSPACENAME # Default smartblob space for use by
> the Informix
> # Server. This is used primarily for Informix Server
> # system statistics collection.
> BLOCKTIMEOUT 3600 # Default timeout for system block
> SYSALARMPROGRAM /opt/informix/etc/evidence.sh # System Alarm program
> path
> # Optimization goal: -1 = ALL_ROWS(Default), 0 = FIRST_ROWS
> OPT_GOAL -1
> ALLOW_NEWLINE 0 # embedded newlines(Yes = 1, No = 0 or
> anything but 1)
> #
> # The following are default settings for enabling Java in the
> database.
> # Replace all occurrences of /usr/informix with the value of
> $INFORMIXDIR.
> #VPCLASS jvp,num=1 # Number of JVPs to start with
> JVPJAVAHOME /usr/informix/extend/krakatoa/jre
> # JRE installation root directory
> JVPHOME /usr/informix/extend/krakatoa # Krakatoa installation
> directory
> JVPPROPFILE /usr/informix/extend/krakatoa/.jvpprops # JVP property
> file
> JDKVERSION 1.3 # JDK version supported by this server
> # The path to the JRE libraries relative to JVPJAVAHOME
> JVPJAVALIB /lib/i386/
> # The JRE libraries to use for the Java VM
> JVPJAVAVM hpi:server:verify:java:net:zip:jpeg
> # use JVPARGS to change Java VM configuration
> #To display jni call
> #JVPARGS -verbose:jni
> # Classpath to use upon Java VM start-up (use _g version for
> debugging)
> #JVPCLASSPATH
> /usr/informix/extend/krakatoa/krakatoa_g.jar:/usr/informix/extend/krakatoa/jdbc_g.jar
> JVPCLASSPATH
> /usr/informix/extend/krakatoa/krakatoa.jar:/usr/informix/extend/krakatoa/jdbc.jar
> LTXEHWM 50 #Provided by ISA configuration system
> LTXHWM 40 #Provided by ISA configuration system
> VPCLASS cpu,num=4,noage #Provided by ISA configuration system
> VPCLASS aio,num=20 #Provided by ISA configuration system
> JVPLOGFILE jvp.log # JVP log file.
>
> ------------------------------------------------------------------------
> Do you Yahoo!?
> Get better spam protection with Yahoo! Mail
>

1. I really don´t know. Probably it is because is the default
installation. Which one is the best?
2.
id key addr size ovhd class blkused blkfree
32769 1381451777 50000000 1328726016 255332 R 324376 20
1343529 1381451817 9f32c000 157696000 5416 V 38008 492
1703988 1381451822 a8990000 33554432 1624 V 5773 2419
1736757 1381451823 aa990000 33554432 1624 V 2299 5893
27000881 1381451824 ac990000 33554432 1624 V 6154 2038
27033650 1381451825 ae990000 33554432 1624 V 6154 2038

3. No processes running at the time.
4.
Informix Dynamic Server Version 9.30.UC1 -- On-Line -- Up 10 days
16:00:30 -- 1582656 Kbytes

Userthreads
address flags sessid user tty wait tout locks nreads
nwrites
9f54b018 ---P--D 1 informix - 0 0 0 2955
62250
9f54b618 ---P--F 0 informix - 0 0 0 0
1320048
9f54bc18 ---P--F 0 informix - 0 0 0 0
295137
9f54c218 ---P--F 0 informix - 0 0 0 0
438247
9f54c818 ---P--F 0 informix - 0 0 0 0
752769
9f54ce18 ---P--- 5 informix - 0 0 0 0
2564
9f54d418 ---P--B 6 informix - 0 0 0 75
1364
9f54da18 Y--P--- 320572 consultc CONSUL08 a0544b38 0 1 0 0
9f54e018 Y--P--- 330771 ceafarma FARMAC20 a4f66788 0 1 38 0
9f54e618 ---P--D 9 informix - 0 0 0 0 0
9f54ec18 Y--P--- 320564 audithos AUDUNI02 a20ef4d0 0 1 0 0
9f54f818 Y--P--- 326532 psgeral PSGERA02 a34a8388 0 1 0 0
9f550418 Y--P--- 332880 consultc CONSUL02 a0370400 0 1 42 0
9f550a18 Y--P--- 323570 posto5an POST5B01 a60bd748 0 1 0 0
9f551618 Y--P--- 328370 ressonan RESSON10 a0f51a10 0 1 269 384
9f552e18 Y--P--- 333385 psgeral PSGERA01 a4dfd700 0 1 0 0
9f553418 Y--P--- 331702 fatparti FATPAR03 a0a1f6a8 0 1 18 0
9f553a18 Y--P--- 320530 recep3 RECEP03 a5686700 0 1 0 0
9f554018 Y--P--- 329937 almoxcon FARMAC08 a5d9f0a8 0 1 102 0
9f554618 Y--P--- 332033 posto5an POST5C01 a2e23820 0 1 0 0
9f554c18 Y--P--- 331857 psgeral SECPSG02 a151ab10 0 1 11 0
9f555218 Y--P--- 332756 ceafarma FARMAC04 a6cef3e0 0 1 0 0
9f555e18 Y--P--- 327997 consultc CONSUL04 a4fcb2a0 0 1 6 0
9f556418 Y--P--- 326928 psgeral SECPSG01 a6251960 0 1 3 0
9f556a18 Y--P--- 333272 consultc CONSUL01 a2e76318 0 1 0 0
9f557018 Y--P--- 332943 superint SUPERI01 a62477d8 0 1 0 0
9f558218 Y--P--- 323588 almoxcen ALMOX03 a50229c8 0 1 183 130
9f558818 Y--P--- 330836 farmhosp FARCTI01 a153ab28 0 1 0 0
9f558e18 Y--P--- 329410 fatconve CONVEN14 a7cdfcb8 0 1 12 0
9f559418 Y--P--- 332109 posto3an POSTO3B2 a151a4c0 0 1 0 0
9f559a18 Y--P--- 326081 radicoba PROCTO01 9ffe4a78 0 1 0 0
9f55a018 Y--P--- 325282 posto2an POSTO2B1 a18b3460 0 1 0 0
9f55a618 Y--P--- 333402 almoxcen ALMOX01 a930a600 0 1 31 0
9f55ac18 Y--P--- 330253 raultomo RADIOL07 a7cdf6e0 0 2 0 0
9f55b818 Y--P--- 332860 blocoobs POSTO2B5 a65bb5f8 0 1 81 0
9f55be18 Y--P--- 319439 fatconve CONVEN02 aaacf5d8 0 1 40 0
9f55c418 Y--P--- 328573 almoxcon FARMAC08 a8ceb2e8 0 1 0 0
9f55ca18 Y--P--- 333095 anatomia ANATOMIA a3aa4c48 0 1 0 0
9f55dc18 Y--P--- 333012 anatomia ANATOMIA a3beae68 0 1 17 0
9f55e218 Y--P--- 333342 labanacl LABTIM03 a5380248 0 1 1 0
9f55e818 Y--P--- 331016 posto3an POSTO3B2 a8180830 0 1 0 0
9f55fa18 Y--P--- 330727 ceafarma FARMAC20 a8fc93e0 0 1 1 0
9f560018 Y--P--- 330578 recep1 PORTCO02 a4cf8018 0 1 0 0
9f560c18 Y--P--- 324618 farmhosp CONSIG01 a1cf10b8 0 1 8 0
9f561218 Y--P--- 328991 centrcir BLOCOC04 a17b9b58 0 1 0 0
9f561e18 Y--P--- 330666 blocoobs POSTO2B5 a2eb1ce0 0 1 0 0
9f562a18 Y--P--- 326911 psgeral SECPSG01 a0f94568 0 1 0 0
9f563018 Y--P--- 332454 cpd ENDOSC02 a52523d8 0 1 1 0
9f563618 Y--P--- 321181 hemodina HEMODI01 a8fc9dd0 0 1 1 0
9f563c18 Y--P--- 333126 cardiolo CARDIO03 a5dd10b8 0 1 17 0
9f564218 Y--P--- 332334 recep1 PORTCO01 a60515b8 0 1 0 0
9f564e18 Y--P--- 333150 posto3an POST3B1 a3f65b80 0 1 0 0
9f565418 Y--P--- 333298 psgeral PSGERA03 a5ead1f8 0 1 0 0
9f565a18 Y--P--- 333269 raultomo RADIOL01 a48deca8 0 1 0 0
9f566018 Y--P--- 324583 almoxcon CONSIG05 a4b5f7a8 0 1 5 0
9f566618 Y--P--- 332398 farmhosp FARMAC30 a5b8cc18 0 1 0 0
9f566c18 Y--P--- 331725 raultomo RADIOL07 a3d23760 0 1 0 0
9f567218 Y--P--- 329750 farmhosp FARMAC35 a3add158 0 1 0 0
9f567818 Y--P--- 333270 recep3 RECEP03 a6051a68 0 1 0 0
9f567e18 Y--P--- 326560 endoscop ENDOSC01 a347baa8 0 1 18 0
9f568a18 Y--P--- 320268 psortope AMBORT01 a43e12c8 0 1 0 0
9f569018 Y--P--- 331731 raultomo RADIOL07 a6d82c80 0 1 92 0
9f569c18 Y--P--- 321809 posto5an POST5A01 a310b270 0 1 135 0
9f56a218 Y--P--- 331200 almoxcon CONSIG05 a29ae938 0 1 0 0
9f56ae18 Y--P--- 330888 posto3an POSTO3B2 a9594ef0 0 1 0 0
9f56b418 Y--P--- 333411 fatconve CONVEN09 a4cf8e70 0 1 1 0
9f56ba18 Y--P--- 331867 psgeral PSGERA03 a5aa7f00 0 1 2 0
9f56c018 Y--P--- 333277 ceafarma FARMAC06 a4b7fdd0 0 1 7 0
9f56c618 Y--P--- 332593 contrece OTORRI01 a3767ac8 0 1 0 0
9f56cc18 Y--P--- 330576 recep1 PORTCO02 a716d680 0 1 0 0
9f56d218 Y--P--- 332667 cpd CPDHFR82 a5aa7158 0 1 1837

=== message truncated ===begin:vcard
fn

aniel Castro
n:Castro;Daniel
email;internet:daniel.castro@feliciorocho.org.br
tel;work:55 31 3339-7361
tel;cell:55 31 9132-7018
version:2.1
end:vcard

---------------------------------
Do you Yahoo!?
Get better spam protection with Yahoo! Mail
--0-770701732-1077895028=:2607
Content-Type: text/html; charset=us-ascii

<DIV>Daniel, </DIV>
<DIV> </DIV>
<DIV>Here are the answers by the topic number that u are having problems with.  </DIV>
<DIV> </DIV>
<DIV>1.  The raw devices is the best because the IO is faster on raw devices.  Have your sysadmin create rawdevices for you.  (Informix 9.4 allows chunks to be created larger than two gigs, so please upgrade to 9.4 to take advantage of large chunks ).</DIV>
<DIV>2.  Ok. onstat -g seg shows that your IDS is dynamically allocating virtual segments !! This not good.  Increase your SHMVRTSIZE.</DIV>
<DIV>4.  Ok, your onstat -u shows that the processes are not waiting for buffers, which is a good thing.  10.  Ok, HERE IS YOUR PROBLEM !  Your IDS is stuck at the check point, this means you are trying to write a whole bunch of stuff to disk at the check point and when this happens IDS has to slow down because it has to write all the data into the disk.  Now the solution for this is to increase your LRU writes.  In order to do this you must know how to tune LRU_MAX_DIRTY and LRU_MIN_DIRTY parameters.  Before you mess with those parameters please read about them and test them in the test box and get a handle of how it works.  I can guarantee you that your checkpoint duration is very long.  You can check that on your online.log.  The 2 most important commands to figure out how to tune these are onstat -F and onstat -R.  The outputs from them will give u an idea how your LRUs and Cleaners are
behaving.  Also, you are doing your check point in every 10 minutes, try to do it every 5 minutes or less and test to see how the engine reacts.  When u wait for a whole 10 minutes the IDS may have gathered a whole bunch of stuff to write to disk.  </DIV>
<DIV> </DIV>
<DIV>10.  Try not to use the default.  Edit your onconfig and enter NETTYPE on it. </DIV>
<DIV>13.  This will give u more cleaners to do the work.  Cleaners do their work and go to sleep.  More Cleaners u have is the better but don't go haywire on that.  Increase is gradually and monitor !</DIV>
<DIV>14.  sar 2 20 will execute sar in every 2 seconds for 20 times.  If the %wio column of the output of this command shows you high numbers (50,80,90) that means you have to tune your OS, your OS is waiting for blocks, I don't know much sysadmin stuff, but know enough to say thats a big problem.</DIV>
<DIV>17.  Ok, ipcs is important !  This will show you what memory segments are allocated to informix from the OS.  Now, you need to learn how to map these memory segments to Informix onstat -g seg output, another words how to map them to your Informix server.  Now, you have to be very carefull, if you are running more than one Informix instance on this machine, then you need to map them to all those instances and figure out which ones are useless and get rid of the ones that are useless.</DIV>
<DIV>18.  Yes, this is another area you should only mess with after reading about it and knowing how they work.  Your /etc/system will show u the configuration.  Increase the sem* parameters in this file gradually and test in the test machine.</DIV>
<DIV> </DIV>
<DIV>I know this is too much.  Everyone, please chip in areas I may have explained vaguely.</DIV>
<DIV> </DIV>
<DIV>Hope this helps.</DIV>
<DIV> </DIV>
<DIV>Ravi Thero. Daniel Castro <daniel.castro@feliciorocho.org.br> wrote:</DIV>
<BLOCKQUOTE class=replbq style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #1010ff 2px solid">Ravi T wrote: > You seem to have many misconfigurations on your Informix server !!! > > Start investigating the following areas ! > > 1. Why are u using cooked files ? > 2. How does your onstat -g seg looks like when the slow down happens ? > 3. Check all the processes that are running when the slow down > happens. Backups, update statistics, batch jobs etc etc. > 4. How does the onstat -u look like when the slow down happens. Are > there processes waiting for buffers ? > 5. What kind of locking are you using ? Page level locks, row locks etc. > 6. When users complain of slowness you need to see what they are > trying to do and check if the tables they are trying to hit are locked > at the time. > 7. When the slowness happens are there logical logs that have not > been backep up
? > 8. Did you change any /etc/system parameters ? If so did u reboot > the Unix server after doing so ? > 9. How does your onstat -g ntt look like ? Does it show processes > that are very old and just sitting there ? > 10. What kind of connections are you using and how many ? Is your > Nettype paramer configured correctly ? By the way you don't even show > NETTYPE parameter on your onconfig file that you have pasted here. > 11. Have you tried the add extra CPUVPS dynamically when slowness > happens. > 12. Is your TEMP dpspace big enough ? Can you add more TEMP > spaces and monitor the system. Its better to have several TEMP spaces > rather than having one big TEMP space. > 13. Increase the CLEANERS to 5 or 6 and monitor the system. > 14. From the OS side when you issue sar command does it show that OS > is waiting for block writes ? > 15. Check onstat -g dic, mem, stk, ses to
figure out the memory and > caching by process. > 16. Reset your profile counts using onstat -z and monitor onstat -p > 17. Use ipcs to lookat memory from OS side and see if there is memory > allocated other stuff other than Informix and investigate. > 18. See if u are allocating enough semophores at the IDS start. > > Those abvoe thoughts are some things to check to start with. > > Ravi Thero > > */Daniel Castro <DANIEL.CASTRO@FELICIOROCHO.ORG.BR>/* wrote: > > Hi, > > I am having informix performance problems. I don´t know where to > start. > Sometimes the database hangs, always from 00:00am to 5:00 am, but it > gives no error in the log file. Can you help me? > Sometimes the users says that the system is slow, but I can´t identify > why. > First, the result of onstat -p: > > Informix Dynamic Server Version 9.30.UC1 -- On-Line -- Up 9
days > 21:06:09 -- 1582656 Kbytes > > Profile > dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached > 67223627 173897813 2077681908 96.76 12042759 61104633 222993237 > 94.60 > > isamtot open start read write rewrite delete commit > rollbk > 2790216351 184683421 616967012 4148277112 65017319 12219659 28562330 > 8807460 884 > > gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs > 0 0 0 0 0 0 0 > > ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes > 7 0 0 336038.88 116702.45 1385 4099 > > bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress > seqscans > 7192942 14028 903998567 4 0 6224 6824574 > 39496093 > > ixda-RA idx-RA da-RA RA-pgsused lchwaits > 12311760 5075181 15389056 32074507 56519934 > > Second my onconfig file: > >
#************************************************* ************************* > # > # INFORMIX SOFTWARE, INC. > # > # Title: onconfig.producao > # Description: Informix Dynamic Server Configuration Parameters > # > #************************************************* ************************* > # Root Dbspace Configuration > ROOTNAME rootdbs # Root dbspace name > ROOTPATH /cooked_file/dbs_rootP # Path for device containing > root dbspace > ROOTOFFSET 0 # Offset of root dbspace into device > (Kbytes) > ROOTSIZE 471039 # Size of root dbspace (Kbytes) > # Disk Mirroring Configuration Parameters > MIRROR 0 # Mirroring flag (Yes = 1, No = 0) > MIRRORPATH # Path for device containing mirrored > root > MIRROROFFSET 0 # Offset into mirrored device (Kbytes) > # Physical Log Configuration > PHYSDBS wpdphlog # Location (dbspace) of physical log > PHYSFILE 60000 #
Physical log file size (Kbytes) > # Logical Log Configuration > LOGFILES 40 # Number of logical log files > LOGSIZE 10000 # Logical log size (Kbytes) > # Diagnostics > MSGPATH /opt/informix/producao.log # System message log file > path > CONSOLE /dev/console # System console message path > ALARMPROGRAM /opt/informix/etc/log_full.sh # Alarm program path > TBLSPACE_STATS 1 # Maintain tblspace statistics > # System Archive Tape Device > TAPEDEV /dev/st0 # Tape device path > TAPEBLK 16 # Tape block size (Kbytes) > TAPESIZE 100000000 # Maximum amount of data to put on > tape (Kbytes) > # Log Archive Tape Device > LTAPEDEV /dev/st0 # Log tape device path > LTAPEBLK 16 # Log tape block size (Kbytes) > LTAPESIZE 100000000 # Max amount of data to put on log > tape (Kbytes) > # Optical > STAGEBLOB # Informix Dynamic Server staging area > # System
Configuration > SERVERNUM 1 # Unique id corresponding to a OnLine > instance > DBSERVERNAME producao # Name of default database server > DBSERVERALIASES # List of alternate dbservernames > DEADLOCK_TIMEOUT 60 # Max time to wait of lock in > distributed env. > RESIDENT 0 # Forced residency flag (Yes = 1, No = > 0) > MULTIPROCESSOR 1 # 0 for single-processor, 1 for > multi-processor > # NUMCPUVPS 1 # Number of user (cpu) vps > SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps > to one > # NOAGE 0 # Process aging > # AFF_SPROC 0 # Affinity start processor > # AFF_NPROCS 0 # Affinity number of processors > # Shared Memory Parameters > LOCKS 200000 # Maximum number of locks > BUFFERS 600000 # Maximum number of shared buffers > # NUMAIOVPS # Number of IO vps > PHYSBUFF 128 # Physical log buffer size (Kbytes) > LOGBUFF 128 # Logical log buffer size
(Kbytes) > CLEANERS 4 # Number of buffer cleaner processes > SHMBASE 0x50000000 # Shared memory base address > SHMVIRTSIZE 154000 # initial virtual shared memory > segment size > SHMADD 32768 # Size of new shared memory segments > (Kbytes) > SHMTOTAL 0 # Total shared memory (Kbytes). > 0=>unlimited > CKPTINTVL 600 # Check point interval (in sec) > LRUS 64 # Number of LRU queues > LRU_MAX_DIRTY 3 # LRU percent dirty begin cleaning > limit > LRU_MIN_DIRTY 1 # LRU percent dirty end cleaning limit > TXTIMEOUT 0x12c # Transaction timeout (in sec) > STACKSIZE 32 # Stack size (Kbytes) > # System Page Size > # BUFFSIZE - OnLine no longer supports this configuration parameter. > # To determine the page size used by OnLine on your > platform > # see the last line of output from the command, 'onstat > -b'. > # Recovery Variables > #
OFF_RECVRY_THREADS: > # Number of parallel worker threads during fast recovery or an offline > restore. > # ON_RECVRY_THREADS: > # Number of parallel worker threads during an online restore. > OFF_RECVRY_THREADS 10 # Default number of offline worker > threads > ON_RECVRY_THREADS 1 # Default number of online worker > threads > # Data Replication Variables > DRINTERVAL 30 # DR max time between DR buffer > flushes (in sec) > DRTIMEOUT 30 # DR network timeout (in sec) > DRLOSTFOUND /usr/informix/etc/dr.lostfound # DR lost+found file > path > # CDR Variables > CDR_EVALTHREADS 1,2 # evaluator threads > (per-cpu-vp,additional) > CDR_DSLOCKWAIT 5 # DS lockwait timeout (seconds) > CDR_QUEUEMEM 4096 # Maximum amount of memory for any CDR > queue (Kbytes) > CDR_NIFCOMPRESS 0 # Link level compression (-1 never, 0 > none, 9 max) > CDR_SERIAL 0,0 # Serial
Column Sequence > CDR_DBSPACE # dbspace for syscdr database > CDR_QHDR_DBSPACE # CDR queue dbspace (default same as > catalog) > CDR_QDATA_SBSPACE # CDR queue smart blob space > CDR_QDATA_SBFLAGS 0 # Log/no-log (default no log) > # Backup/Restore variables > BAR_ACT_LOG /home/informix/bar_act.log > # ON-Bar Log file - not in /tmp please > BAR_DEBUG_LOG /home/informix/bar_dbug.log > # ON-Bar Debug Log - not in /tmp > please > BAR_MAX_BACKUP 0 > BAR_RETRY 1 > BAR_NB_XPORT_COUNT 10 > BAR_XFER_BUF_SIZE 31 > RESTARTABLE_RESTORE on > BAR_PROGRESS_FREQ 0 > # Informix Storage Manager variables > ISM_DATA_POOL ISMData > ISM_LOG_POOL ISMLogs > # Read Ahead Variables > RA_PAGES 32 # Number of pages to attempt to read > ahead > RA_THRESHOLD 20 # Number of pages left before next > group > # DBSPACETEMP: > # OnLine equivalent of DBTEMP
for SE. This is the list of dbspaces > # that the OnLine SQL Engine will use to create temp tables etc. > # If specified it must be a colon separated list of dbspaces that > exist > # when the OnLine system is brought online. If not specified, or if > # all dbspaces specified are invalid, various ad hoc queries will > create > # temporary files in /tmp instead. > DBSPACETEMP wpdtemp1 # Default temp dbspaces > # DUMP*: > # The following parameters control the type of diagnostics information > which > # is preserved when an unanticipated error condition (assertion > failure) occurs > # during OnLine operations. > # For DUMPSHMEM, DUMPGCORE and DUMPCORE 1 means Yes, 0 means No. > DUMPDIR /tmp # Preserve diagnostics in this > directory > DUMPSHMEM 1 # Dump a copy of shared memory > DUMPGCORE 0 # Dump a core image using 'gcore' > DUMPCORE 0 # Dump a core image
(Warning:this > aborts OnLine) > DUMPCNT 1 # Number of shared memory or gcore > dumps for > # a single user's session > FILLFACTOR 90 # Fill factor for building indexes > # method for OnLine to use when determining current time > USEOSTIME 0 # 0: use internal time(fast), 1: get > time from OS(slow) > # Parallel Database Queries (pdq) > MAX_PDQPRIORITY 100 # Maximum allowed pdqpriority > DS_MAX_QUERIES # Maximum number of decision support > queries > DS_TOTAL_MEMORY # Decision support memory (Kbytes) > DS_MAX_SCANS 1048576 # Maximum number of decision support > scans > DATASKIP off # List of dbspaces to skip > # OPTCOMPIND > # 0 => Nested loop joins will be preferred (where > # possible) over sortmerge joins and hash joins. > # 1 => If the transaction isolation mode is not > # "repeatable read", optimizer behaves as in (2) > # below. Otherwise it
behaves as in (0) above. > # 2 => Use costs regardless of the transaction isolation > # mode. Nested loop joins are not necessarily > # preferred. Optimizer bases its decision purely > # on costs. > OPTCOMPIND 2 # To hint the optimizer > DIRECTIVES 1 # Optimizer DIRECTIVES ON (1/Default) > or OFF (0) > ONDBSPACEDOWN 2 # Dbspace down option: 0 = CONTINUE, 1 > = ABORT, 2 = WAIT > OPCACHEMAX 0 # Maximum optical cache size (Kbytes) > # HETERO_COMMIT (Gateway participation in distributed transactions) > # 1 => Heterogeneous Commit is enabled > # 0 (or any other value) => Heterogeneous Commit is disabled > HETERO_COMMIT 0 > SBSPACENAME # Default smartblob space name - this > is where blobs > # go if no sbspace is specified when the smartblob is > # created. It is also used by some datablades as > # the location to put their smartblobs. > SYSSBSPACENAME # Default
smartblob space for use by > the Informix > # Server. This is used primarily for Informix Server > # system statistics collection. > BLOCKTIMEOUT 3600 # Default timeout for system block > SYSALARMPROGRAM /opt/informix/etc/evidence.sh # System Alarm program > path > # Optimization goal: -1 = ALL_ROWS(Default), 0 = FIRST_ROWS > OPT_GOAL -1 > ALLOW_NEWLINE 0 # embedded newlines(Yes = 1, No = 0 or > anything but 1) > # > # The following are default settings for enabling Java in the > database. > # Replace all occurrences of /usr/informix with the value of > $INFORMIXDIR. > #VPCLASS jvp,num=1 # Number of JVPs to start with > JVPJAVAHOME /usr/informix/extend/krakatoa/jre > # JRE installation root directory > JVPHOME /usr/informix/extend/krakatoa # Krakatoa installation > directory > JVPPROPFILE /usr/informix/extend/krakatoa/.jvpprops # JVP property >
file > JDKVERSION 1.3 # JDK version supported by this server > # The path to the JRE libraries relative to JVPJAVAHOME > JVPJAVALIB /lib/i386/ > # The JRE libraries to use for the Java VM > JVPJAVAVM hpi:server:verify:java:net:zip:jpeg > # use JVPARGS to change Java VM configuration > #To display jni call > #JVPARGS -verbose:jni > # Classpath to use upon Java VM start-up (use _g version for > debugging) > #JVPCLASSPATH > /usr/informix/extend/krakatoa/krakatoa_g.jar:/usr/informix/extend/krakatoa/jdbc_g.jar > JVPCLASSPATH > /usr/informix/extend/krakatoa/krakatoa.jar:/usr/informix/extend/krakatoa/jdbc.jar > LTXEHWM 50 #Provided by ISA configuration system > LTXHWM 40 #Provided by ISA configuration system > VPCLASS cpu,num=4,noage #Provided by ISA configuration system > VPCLASS aio,num=20 #Provided by ISA configuration system > JVPLOGFILE jvp.log # JVP log
file. > > ------------------------------------------------------------------------ > Do you Yahoo!? > Get better spam protection with Yahoo! Mail > <HTTP: tools antispam.yahoo.com *http: mailtag_us us.rd.yahoo.com> 1. I really don´t know. Probably it is because is the default installation. Which one is the best? 2. id key addr size ovhd class blkused blkfree 32769 1381451777 50000000 1328726016 255332 R 324376 20 1343529 1381451817 9f32c000 157696000 5416 V 38008 492 1703988 1381451822 a8990000 33554432 1624 V 5773 2419 1736757 1381451823 aa990000 33554432 1624 V 2299 5893 27000881 1381451824 ac990000 33554432 1624 V 6154 2038 27033650 1381451825 ae990000 33554432 1624 V 6154 2038 3. No processes running at the time. 4. Informix Dynamic Server Version 9.30.UC1 -- On-Line -- Up 10 days 16:00:30 -- 1582656 Kbytes Userthreads address flags sessid user tty wait tout locks nreads
 nwrites 9f54b018 ---P--D 1 informix - 0 0 0 2955 62250 9f54b618 ---P--F 0 informix - 0 0 0 0 1320048 9f54bc18 ---P--F 0 informix - 0 0 0 0 295137 9f54c218 ---P--F 0 informix - 0 0 0 0 438247 9f54c818 ---P--F 0 informix - 0 0 0 0 752769 9f54ce18 ---P--- 5 informix - 0 0 0 0 2564 9f54d418 ---P--B 6 informix - 0 0 0 75 1364 9f54da18 Y--P--- 320572 consultc CONSUL08 a0544b38 0 1 0 0 9f54e018 Y--P--- 330771 ceafarma FARMAC20 a4f66788 0 1 38 0 9f54e618 ---P--D 9 informix - 0 0 0 0 0 9f54ec18 Y--P--- 320564 audithos AUDUNI02 a20ef4d0 0 1 0 0 9f54f818 Y--P--- 326532 psgeral PSGERA02 a34a8388 0 1 0 0 9f550418 Y--P--- 332880 consultc CONSUL02 a0370400 0 1 42 0 9f550a18 Y--P--- 323570 posto5an POST5B01 a60bd748 0 1 0 0 9f551618 Y--P--- 328370 ressonan RESSON10 a0f51a10 0 1 269 384 9f552e18 Y--P--- 333385 psgeral PSGERA01 a4dfd700 0 1 0 0 9f553418 Y--P--- 331702 fatparti FATPAR03 a0a1f6a8 0 1 18 0 9f553a18
Y--P--- 320530 recep3 RECEP03 a5686700 0 1 0 0 9f554018 Y--P--- 329937 almoxcon FARMAC08 a5d9f0a8 0 1 102 0 9f554618 Y--P--- 332033 posto5an POST5C01 a2e23820 0 1 0 0 9f554c18 Y--P--- 331857 psgeral SECPSG02 a151ab10 0 1 11 0 9f555218 Y--P--- 332756 ceafarma FARMAC04 a6cef3e0 0 1 0 0 9f555e18 Y--P--- 327997 consultc CONSUL04 a4fcb2a0 0 1 6 0 9f556418 Y--P--- 326928 psgeral SECPSG01 a6251960 0 1 3 0 9f556a18 Y--P--- 333272 consultc CONSUL01 a2e76318 0 1 0 0 9f557018 Y--P--- 332943 superint SUPERI01 a62477d8 0 1 0 0 9f558218 Y--P--- 323588 almoxcen ALMOX03 a50229c8 0 1 183 130 9f558818 Y--P--- 330836 farmhosp FARCTI01 a153ab28 0 1 0 0 9f558e18 Y--P--- 329410 fatconve CONVEN14 a7cdfcb8 0 1 12 0 9f559418 Y--P--- 332109 posto3an POSTO3B2 a151a4c0 0 1 0 0 9f559a18 Y--P--- 326081 radicoba PROCTO01 9ffe4a78 0 1 0 0 9f55a018 Y--P--- 325282 posto2an POSTO2B1 a18b3460 0 1 0 0 9f55a618 Y--P--- 333402 almoxcen ALMOX01 a930a600 0 1 31 0 9f55ac18
Y--P--- 330253 raultomo RADIOL07 a7cdf6e0 0 2 0 0 9f55b818 Y--P--- 332860 blocoobs POSTO2B5 a65bb5f8 0 1 81 0 9f55be18 Y--P--- 319439 fatconve CONVEN02 aaacf5d8 0 1 40 0 9f55c418 Y--P--- 328573 almoxcon FARMAC08 a8ceb2e8 0 1 0 0 9f55ca18 Y--P--- 333095 anatomia ANATOMIA a3aa4c48 0 1 0 0 9f55dc18 Y--P--- 333012 anatomia ANATOMIA a3beae68 0 1 17 0 9f55e218 Y--P--- 333342 labanacl LABTIM03 a5380248 0 1 1 0 9f55e818 Y--P--- 331016 posto3an POSTO3B2 a8180830 0 1 0 0 9f55fa18 Y--P--- 330727 ceafarma FARMAC20 a8fc93e0 0 1 1 0 9f560018 Y--P--- 330578 recep1 PORTCO02 a4cf8018 0 1 0 0 9f560c18 Y--P--- 324618 farmhosp CONSIG01 a1cf10b8 0 1 8 0 9f561218 Y--P--- 328991 centrcir BLOCOC04 a17b9b58 0 1 0 0 9f561e18 Y--P--- 330666 blocoobs POSTO2B5 a2eb1ce0 0 1 0 0 9f562a18 Y--P--- 326911 psgeral SECPSG01 a0f94568 0 1 0 0 9f563018 Y--P--- 332454 cpd ENDOSC02 a52523d8 0 1 1 0 9f563618 Y--P--- 321181 hemodina HEMODI01 a8fc9dd0 0 1 1 0 9f563c18 Y--P---
333126 cardiolo CARDIO03 a5dd10b8 0 1 17 0 9f564218 Y--P--- 332334 recep1 PORTCO01 a60515b8 0 1 0 0 9f564e18 Y--P--- 333150 posto3an POST3B1 a3f65b80 0 1 0 0 9f565418 Y--P--- 333298 psgeral PSGERA03 a5ead1f8 0 1 0 0 9f565a18 Y--P--- 333269 raultomo RADIOL01 a48deca8 0 1 0 0 9f566018 Y--P--- 324583 almoxcon CONSIG05 a4b5f7a8 0 1 5 0 9f566618 Y--P--- 332398 farmhosp FARMAC30 a5b8cc18 0 1 0 0 9f566c18 Y--P--- 331725 raultomo RADIOL07 a3d23760 0 1 0 0 9f567218 Y--P--- 329750 farmhosp FARMAC35 a3add158 0 1 0 0 9f567818 Y--P--- 333270 recep3 RECEP03 a6051a68 0 1 0 0 9f567e18 Y--P--- 326560 endoscop ENDOSC01 a347baa8 0 1 18 0 9f568a18 Y--P--- 320268 psortope AMBORT01 a43e12c8 0 1 0 0 9f569018 Y--P--- 331731 raultomo RADIOL07 a6d82c80 0 1 92 0 9f569c18 Y--P--- 321809 posto5an POST5A01 a310b270 0 1 135 0 9f56a218 Y--P--- 331200 almoxcon CONSIG05 a29ae938 0 1 0 0 9f56ae18 Y--P--- 330888 posto3an POSTO3B2 a9594ef0 0 1 0 0 9f56b418 Y--P--- 333411
fatconve CONVEN09 a4cf8e70 0 1 1 0 9f56ba18 Y--P--- 331867 psgeral PSGERA03 a5aa7f00 0 1 2 0 9f56c018 Y--P--- 333277 ceafarma FARMAC06 a4b7fdd0 0 1 7 0 9f56c618 Y--P--- 332593 contrece OTORRI01 a3767ac8 0 1 0 0 9f56cc18 Y--P--- 330576 recep1 PORTCO02 a716d680 0 1 0 0 9f56d218 Y--P--- 332667 cpd CPDHFR82 a5aa7158 0 1 1837 === message truncated ===begin:vcard fn

aniel Castro n:Castro;Daniel email;internet:daniel .castro@feliciorocho.org.br tel;work:55 31 3339-7361 tel;cell:55 31 9132-7018 version:2.1 end:vcard </BLOCKQUOTE><hr SIZE=1>
Do you Yahoo!? 
Get better spam protection with <a href="http://us.rd.yahoo.com/mailtag_us/*http://antispam.yahoo.com/tools">Yahoo! Mail</a>
--0-770701732-1077895028=:2607--
sending to informix-list

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode