I have been gathering information about this system. At the end
really strange things end up not being true. This is another time it
happens.

Though the people that were responsible of this before -and supposed
to be knowledgeable in Informix- told me the system I posted the other day
was not good, I have been doing some testing using LoadRunner and I found
what I was told do not seem to be right.

At the end, all the reporting functionality in the application goes
against another Informix instance. We could say the "online" functionality
goes against the Informix I posted the other day and that "reports" go
against the Informix instance that shows poor performance. Between these two
systems there is transactional replication, which I do not know very well in
Informix.

The onconfig for this system looks like:

#************************************************* *************************
#
# INFORMIX SOFTWARE, INC.
#
# Title: onconfig.std
# Description: Informix Dynamic Server Configuration Parameters
#
#************************************************* *************************

# Root Dbspace Configuration

ROOTNAME rootdbs # Root dbspace name
ROOTPATH X:\IFMXDATA\ol_espmadinformix2\rootdbs_dat.000
#
Path for device containing root dbspace
ROOTOFFSET 0 # Offset of root dbspace into device (Kbytes)
ROOTSIZE 30720 # Size of root dbspace (Kbytes)

# Disk Mirroring Configuration Parameters

MIRROR 0 # Mirroring flag (Yes = 1, No = 0)
MIRRORPATH # Path for device containing mirrored root
MIRROROFFSET 0 # Offset into mirrored device (Kbytes)

# Physical Log Configuration

PHYSDBS physdbs # Location (dbspace) of physical log
PHYSFILE 80000 # Physical log file size (Kbytes)

# Logical Log Configuration

LOGFILES 10 # Number of logical log files
LOGSIZE 1500 # Logical log size (Kbytes)
LOG_BACKUP_MODE CONT # Logical log backup mode (MANUAL, CONT)

# Diagnostics

MSGPATH X:\IFMXDATA\ol_replica.log # System message log file path
CONSOLE X:\IFMXDATA\conol_replica.log # System console message path

# To automatically backup logical logs, edit alarmprogram.bat and set
# BACKUPLOGS=Y
ALARMPROGRAM E:\informix\etc\alarmprogram.bat # Alarm
program path
TBLSPACE_STATS 1 # Maintain tblspace statistics


# System Diagnostic Script.
# SYSALARMPROGRAM - Full path of the system diagnostic script (e.g.
# c:\informix\etc\evidence.bat.) Set this parameter
# if you want a different Diagnostic Script than
# {INFORMIXDIR}\etc\evidence.bat, which is default.

# System Archive Tape Device

#TAPEDEV nul # Tape device path
TAPEDEV L:\copia # Tape device path
TAPEBLK 16 # Tape block size (Kbytes)
TAPESIZE 75000000 # Maximum amount of data to put on tape (Kbytes)

# Log Archive Tape Device

LTAPEDEV X:\copilog # Log tape device path
LTAPEBLK 16 # Log tape block size (Kbytes)
LTAPESIZE 80000000 # Max amount of data to put on log tape (Kbytes)

# Optical

STAGEBLOB # Informix Dynamic Server/Optical staging
area
OPTICAL_LIB_PATH # Location of Optical Subsystem driver DLL

# System Configuration

SERVERNUM 1 # Unique id corresponding to a OnLine instance
DBSERVERNAME ol_replica # Name of default database server
DBSERVERALIASES # List of alternate dbservernames
NETTYPE soctcp,2,500,NET # Override sqlhosts nettype parameters
DEADLOCK_TIMEOUT 90 # Max time to wait of lock in distributed env.
#RESIDENT 0 # Forced residency flag (Yes = 1, No = 0)
RESIDENT 1 # Forced residency flag (Yes = 1, No = 0)

MULTIPROCESSOR 1 # 0 for single-processor, 1 for
multi-processor
NUMCPUVPS 2 # Number of user (cpu) vps
SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps to
one

NOAGE 0 # Process aging
AFF_SPROC 0 # Affinity start processor
AFF_NPROCS 0 # Affinity number of processors

# Shared Memory Parameters

LOCKS 500000 # Maximum number of locks
#BUFFERS 200000 # Maximum number of shared buffers
BUFFERS 300000 # Maximum number of shared buffers
NUMAIOVPS 1 # Number of IO vps
PHYSBUFF 128 # Physical log buffer size (Kbytes)
#LOGBUFF 64 # Logical log buffer size (Kbytes)
LOGBUFF 16 # Logical log buffer size (Kbytes)
CLEANERS 8 # Number of buffer cleaner processes
SHMBASE 0xC000000L # Shared memory base address
#SHMVIRTSIZE 128000 # initial virtual shared memory segment size
SHMVIRTSIZE 256000 # initial virtual shared memory segment size
SHMADD 32768 # Size of new shared memory segments
(Kbytes)
SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited
CKPTINTVL 300 # Check point interval (in sec)
#LRUS 64 # Number of LRU queues
LRUS 256 # Number of LRU queues
LRU_MAX_DIRTY 2 # LRU percent dirty begin cleaning limit
LRU_MIN_DIRTY 1 # LRU percent dirty end cleaning limit
TXTIMEOUT 300 # Transaction timeout (in sec)
STACKSIZE 64 # Stack size (Kbytes)

# Dynamic Logging
# DYNAMIC_LOGS:
# 2 : server automatically add a new logical log when necessary. (ON)
# 1 : notify DBA to add new logical logs when necessary. (ON)
# 0 : cannot add logical log on the fly. (OFF)
#
# When dynamic logging is on, we can have higher values for LTXHWM/LTXEHWM,
# because the server can add new logical logs during long transaction
rollback.
# However, to limit the number of new logical logs being added,
LTXHWM/LTXEHWM
# can be set to smaller values.
#
# If dynamic logging is off, LTXHWM/LTXEHWM need to be set to smaller values
# to avoid long transaction rollback hanging the server due to lack of
logical
# log space, i.e. 50/60 or lower.

DYNAMIC_LOGS 0
LTXHWM 45
LTXEHWM 50

# System Page Size
# BUFFSIZE - Dynamic Server no longer supports this configuration parameter.
# To determine the page size used by Dynamic Server on your
platform
# see the last line of output from the command, 'onstat -b'.


# Recovery Variables
# OFF_RECVRY_THREADS:
# Number of parallel worker threads during fast recovery or an offline
restore.
# ON_RECVRY_THREADS:
# Number of parallel worker threads during an online restore.

OFF_RECVRY_THREADS 10 # Default number of offline worker threads
ON_RECVRY_THREADS 1 # Default number of online worker threads

# Data Replication Variables
DRINTERVAL 30 # DR max time between DR buffer flushes (in sec)
DRTIMEOUT 30 # DR network timeout (in sec)
DRLOSTFOUND E:\INFORMIX\tmp # DR lost+found file path

# CDR Variables
CDR_EVALTHREADS 1,2 # evaluator threads (per-cpu-vp,additional)
CDR_DSLOCKWAIT 5 # DS lockwait timeout (seconds)
CDR_QUEUEMEM 4096 # Maximum memory for any CDR queue (Kbytes)
CDR_QHDR_DBSPACE # CDR queue dbspace (default same as catalog)
CDR_QDATA_SBSPACE # CDR queue smart blob space
CDR_QDATA_SBFLAGS 2 # Log/no-log (default no log)
CDR_NIFCOMPRESS 0 # Link level compression (-1 never, 0 none, 9 max)

# Backup/Restore variables
BAR_ACT_LOG e:\informix\bar_act.log
BAR_MAX_BACKUP 0
BAR_RETRY 1
BAR_NB_XPORT_COUNT 10
BAR_XFER_BUF_SIZE 15
BAR_BSALIB_PATH E:\ISM\2.20\bin\libbsa.dll #Location of ISM XBSA dll

RESTARTABLE_RESTORE ON #To support restartable restore..values on/off

# Informix Storage Manager variables
ISM_DATA_POOL ISMData
ISM_LOG_POOL ISMLogs

# Read Ahead Variables
RA_PAGES 32 # Number of pages to attempt to read ahead
RA_THRESHOLD 30 # Number of pages left before next group

# DBSPACETEMP:
# Dynamic Server equivalent of DBTEMP for SE. This is the list of dbspaces
# that the Dynamic Server SQL Engine will use to create temp tables etc.
# If specified it must be a colon separated list of dbspaces that exist
# when the Dynamic Server system is brought online. If not specified, or if
# all dbspaces specified are invalid, various ad hoc queries will create
# temporary files in /tmp instead.

DBSPACETEMP tempdbs,tempdbs1 # Default temp dbspaces

# DUMP*:
# The following parameters control the type of diagnostics information which
# is preserved when an unanticipated error condition (assertion failure)
occurs
# during Dynamic Server operations.
# For DUMPSHMEM, DUMPGCORE and DUMPCORE 1 means Yes, 0 means No.

DUMPDIR E:\tmp # Preserve diagnostics in this directory
DUMPSHMEM 0 # Dump a copy of shared memory
DUMPGCORE 0 # Dump a core image using 'gcore'
DUMPCORE 0 # Dump a core image (Warning:this aborts Dynamic
Server)
DUMPCNT 1 # Number of shared memory or gcore dumps for
# a single user's session

FILLFACTOR 90 # Fill factor for building indexes

# method for Dynamic Server to use when determining current time
USEOSTIME 0 # 0: use internal time(fast), 1: get time from OS(slow)

# Parallel Database Queries (pdq)
MAX_PDQPRIORITY 10 # Maximum allowed pdqpriority
DS_MAX_QUERIES 32 # Maximum number of decision support queries
DS_TOTAL_MEMORY 4096 # Decision support memory (Kbytes)
DS_MAX_SCANS 1048576 # Maximum number of decision support scans
DATASKIP off
# OPTCOMPIND
# 0 => Nested loop joins will be preferred (where
# possible) over sortmerge joins and hash joins.
# 1 => If the transaction isolation mode is not
# "repeatable read", optimizer behaves as in (2)
# below. Otherwise it behaves as in (0) above.
# 2 => Use costs regardless of the transaction isolation
# mode. Nested loop joins are not necessarily
# preferred. Optimizer bases its decision purely
# on costs.
#OPTCOMPIND 0 # To hint the optimizer
OPTCOMPIND 2 # To hint the optimizer

DIRECTIVES 1 # Optimizer DIRECTIVES ON (1/Default) or OFF (0)

ONDBSPACEDOWN 2 # Dbspace down option: 0 = CONTINUE, 1 = ABORT, 2 =
WAIT

OPCACHEMAX 0 # Maximum optical cache size (Kbytes)

# HETERO_COMMIT (Gateway participation in distributed transactions)
# 1 => Heterogeneous Commit is enabled
# 0 (or any other value) => Heterogeneous Commit is disabled
HETERO_COMMIT 0

SBSPACENAME sbspace # Default smartblob space name

BLOCKTIMEOUT 3600 # Default timeout for system block

# Optimization goal: -1 = ALL_ROWS(Default), 0 = FIRST_ROWS
#OPT_GOAL -1
OPT_GOAL 0

ALLOW_NEWLINE 0 # embedded newlines(Yes = 1, No = 0 or anything but
1)
SYSSBSPACENAME sbspace # Default system smartblob space name

#
# The following are default settings for enabling Java in the database.
#

#VPCLASS jvp,num=1 # Number of JVPs to start with

JVPJAVAHOME # JDK installation root directory
JVPHOME E:\informix\extend\krakatoa # Krakatoa
installation directory

JVPLOGFILE E:\informix\extend\krakatoa\ol_espmadinformix2_jvp .log #
JVP log file.
JVPPROPFILE E:\informix\extend\krakatoa\.jvpprops_ol_espmadinf ormix2 #
JVP property file

JDKVERSION 1.3 # JDK version supported by this server

# The path to the JRE libraries relative to JVPJAVAHOME
JVPJAVALIB \jre\bin\

JVPJAVAVM hpi;server;verify;java;net;zip;jpeg

# Classpath to use upon Java VM start-up (use _g version for debugging)
JVPCLASSPATH

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%
%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%
%%

This system is 100% CPU ALL the time, with 30 to 50 sessions. Some
components deal with reports, which parametrization and calculation go
against this Informix instance. The system is a 2-way Pentim III box with 2
Gb Memory.

The maximum number of connections is 30-35 and the average checkpoint time
is 4-5 seconds and 1 time out of 10 the checkpoint takes 20 to 40 seconds to
take place. Interestingly, most of the access to this instance is read-only
reporting tasks.

This system is booted every morning; the logs looks like:

Tue Dec 21 07:19:57 2004

07:19:57 Booting Language <c> from module <>
07:19:58 Loading Module <CNULL>
07:19:58 Booting Language <builtin> from module <>
07:19:58 Loading Module <BUILTINNULL>
07:20:55 IBM Informix Dynamic Server Version 9.40.TC3 Software Serial
Number AAA#B000000
07:20:56 IBM Informix Dynamic Server Initialized -- Shared Memory
Initialized.

07:20:56 DR: Logs already cleared
07:20:56 Physical Recovery Started at Page (4:110730).
07:20:58 Physical Recovery Complete: 0 Pages Examined, 0 Pages Restored.
07:20:59 DR: Trying to connect to primary server = ol_espmadinformix2
07:20:59 Dataskip is now OFF for all dbspaces
07:20:59 Restartable Restore has been ENABLED
07:20:59 Recovery Mode
07:21:11 DR: Secondary server connected
07:21:12 DR: Secondary server needs failure recovery

07:21:13 DR: Failure recovery from disk in progress ...
07:21:17 Logical Recovery Started.
07:21:17 10 recovery worker threads will be started.
07:21:20 Start Logical Recovery - Start Log 30287, End Log ?
07:21:20 Starting Log Position - 30287 0xf4c018
07:23:06 Checkpoint Completed: duration was 0 seconds.
07:23:07 Checkpoint loguniq 30287, logpos 0xf4d018, timestamp: -809044087


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%
%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%
%%

I attach the same commands you asked me to execute the other day; I will
start having a look at this right now. I think the problem must not be that
much hidden and maybe I can manage to find out.

Many thanks again, Obnoxio. Have a nice day.

Leopold,

If most of your tasks are reporting only and you are using HDR as the
log you posted shows, why not run the read only reporting tasks on the
DR secondary and reduce the load on your primary server?

On a point of information HDR, which you call transactional replication,
is very different to the feature of the same name in MS SQL Server.

To improve performance try setting OPTCOMPIND to 0. It may persuade the
optmiser to use better query plans. Also if you aren't using your table
space statistics set TBLSPACE_STATS to 0.

Ben.