SEO

I have the confusing fact that in contrary to Linux (UnitedLinux or
SuseEnterpriseServer)
I miss some informations I write to a file.
It seems that defining this file to be used for stdout+stderr in a csh-
call like in
myprog ... >>& mylog
prevents later in the callhirarchy,
that is by things called from myprog,
and other scripts running in parallel as well
(because this csh is running with nohup parallel to those)
to put their output explicitly, e.g. by
echo ... >> mylog
into the same file.

Is there some parameter or "set" command to make AIX behave like the
others?
I know that if those messages REALLY come in parallel, there MUST be
some conflict
due to the serialization of the output, so I might get things in a
"wrong" order
(I really don't want to discuss if this application is very
sophisticated
or should be modernized/recoded to a certain extent ;-),
but I miss EVERYTHING of the other processes, not only some of the
letters,
so the question occurs if it's a configurable behavior of AIX?

The highest level of any bos.*-module of the machine in lslpp-output
is 5.3.0.61,
but I think the behaviour hasn't changed and was the same (and always
frustrating) in former releases?

Thanks in advance
bine

On Nov 13, 2:05 am, bine <sabine.hubrig-schaumb...@sungard.de> wrote:
> I have the confusing fact that in contrary to Linux (UnitedLinux or
> SuseEnterpriseServer)
> I miss some informations I write to a file.
> It seems that defining this file to be used for stdout+stderr in a csh-
> call like in
> myprog ... >>& mylog
> prevents later in the callhirarchy,
> that is by things called from myprog,
> and other scripts running in parallel as well
> (because this csh is running with nohup parallel to those)
> to put their output explicitly, e.g. by
> echo ... >> mylog
> into the same file.
>
> Is there some parameter or "set" command to make AIX behave like the
> others?
> I know that if those messages REALLY come in parallel, there MUST be
> some conflict
> due to the serialization of the output, so I might get things in a
> "wrong" order
> (I really don't want to discuss if this application is very
> sophisticated
> or should be modernized/recoded to a certain extent ;-),
> but I miss EVERYTHING of the other processes, not only some of the
> letters,
> so the question occurs if it's a configurable behavior of AIX?
>
> The highest level of any bos.*-module of the machine in lslpp-output
> is 5.3.0.61,
> but I think the behaviour hasn't changed and was the same (and always
> frustrating) in former releases?
>
> Thanks in advance
> bine

I cannot tell by your description whether you do or do not
ultimately require _either_ serialization or just successul
writes to "mylog" of your "other scripts". Only for SGI Irix
can I specifically say that shell-level output redirection
is written line-by-line atomically. For other OSes I cannot
say what the behavior is or should be.

The common wisdom is that t/csh is problematic for scripting
purposes; under AIX and anything goes!

D,"comp.unix.shell -- Csh Programming Considered Harmful",
1993-08-11,http://www.faqs.org/faqs/unix-faq/sh...hynot/;http://
www.cs.uleth.ca/~holzmann/C/shells/csh.avoidit,$0,"webpage","Tom
Christiansen" <tchrist@wraeththu.cs.colorado.edu>

"Csh Programming Considered Harmful"
http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/

I know it's more than you want to do, but for robust and
portable coding, not relying on the underlying I/O of your
OS, you will want to explicitly control process serialization.
The best overall library I have found and used is:

"semaphore.ksh" and "qjob.ksh":
http://www.unixreview.com/documents/...408f/0408f.htm
http://www.samag.com/code/
ftp://ftp.mfi.com/pub/sysadmin/2004/aug2004.zip
Schaefer, Ed and John Spurgeon. "Queuing Jobs with qjob". 2005-07. Sys
Admin. 14(8):XXX. <http://www.unixreview.com/documents/s=9823/sam0508h/
0508h.htm>.

There are a zillion others of inferior quality, but are
perhaps better suited to simpler demands -- just do a search
in C.U.S. or try:

"mklock.sh"
http://www.shelldorado.com/scripts/quickies/mklock
http://www.shelldorado.com/newslette...003-1-Jul.html

"lock_unlock.ksh"
http://www.unixlabplus.com/unix-prog/lock_unlock/
http://www.unixreview.com/documents/s=9040/ur0402g/

.... and so on. Note that absolutely none of them are t/csh
scripts....

=Brian

On 14 Nov., 03:46, bsh <brian_hi...@rocketmail.com> wrote:
> On Nov 13, 2:05 am, bine <sabine.hubrig-schaumb...@sungard.de> wrote:
>
>
>
>
>
> > I have the confusing fact that in contrary to Linux (UnitedLinux or
> > SuseEnterpriseServer)
> > I miss some informations I write to a file.
> > It seems that defining this file to be used for stdout+stderr in a csh-
> > call like in
> > myprog ... >>& mylog
> > prevents later in the callhirarchy,
> > that is by things called from myprog,
> > and other scripts running in parallel as well
> > (because this csh is running with nohup parallel to those)
> > to put their output explicitly, e.g. by
> > echo ... >> mylog
> > into the same file.
>
> > Is there some parameter or "set" command to make AIX behave like the
> > others?
> > I know that if those messages REALLY come in parallel, there MUST be
> > some conflict
> > due to the serialization of the output, so I might get things in a
> > "wrong" order
> > (I really don't want to discuss if this application is very
> > sophisticated
> > or should be modernized/recoded to a certain extent ;-),
> > but I miss EVERYTHING of the other processes, not only some of the
> > letters,
> > so the question occurs if it's a configurable behavior of AIX?
>
> > The highest level of any bos.*-module of the machine in lslpp-output
> > is 5.3.0.61,
> > but I think the behaviour hasn't changed and was the same (and always
> > frustrating) in former releases?
>
> > Thanks in advance
> > bine
>
> I cannot tell by your description whether you do or do not
> ultimately require _either_ serialization or just successul
> writes to "mylog" of your "other scripts". Only for SGI Irix
> can I specifically say that shell-level output redirection
> is written line-by-line atomically. For other OSes I cannot
> say what the behavior is or should be.
>
> The common wisdom is that t/csh is problematic for scripting
> purposes; under AIX and anything goes!
>
> D,"comp.unix.shell -- Csh Programming Considered Harmful",
> 1993-08-11,http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/;http://www.cs.uleth.ca/~holzmann/C/shells/csh.avoidit,$0,"webpage","Tom
> Christiansen" <tchr...@wraeththu.cs.colorado.edu>
>
> "Csh Programming Considered Harmful"http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/
>
> I know it's more than you want to do, but for robust and
> portable coding, not relying on the underlying I/O of your
> OS, you will want to explicitly control process serialization.
> The best overall library I have found and used is:
>
> "semaphore.ksh" and "qjob.ksh":http://www.unixreview.com/documents/...04/aug2004.zip
> Schaefer, Ed and John Spurgeon. "Queuing Jobs with qjob". 2005-07. Sys
> Admin. 14(8):XXX. <http://www.unixreview.com/documents/s=9823/sam0508h/
> 0508h.htm>.
>
> There are a zillion others of inferior quality, but are
> perhaps better suited to simpler demands -- just do a search
> in C.U.S. or try:
>
> "mklock.sh"http://www.shelldorado.com/scripts/quickies/mklockhttp://www.shelldorado.com/newsletter/issues/2003-1-Jul.html
>
> "lock_unlock.ksh"http://www.unixlabplus.com/unix-prog/lock_unlock/http://www.unixreview.com/documents/s=9040/ur0402g/
>
> ... and so on. Note that absolutely none of them are t/csh
> scripts....
>
> =Brian- Zitierten Text ausblenden -
>
> - Zitierten Text anzeigen -

thanks brian,
I do not require absolute serialization of the outputs, I risk to have
some lines mixed, but at the moment i MISS a lot of output which
brought me to the idea that on AIX there might be some locking-
mechanism involved whereras on my other systems I get the lines.
And I have no chance to recode that stuff so I was just looking for
some "kernel"parameter (smit ... whatsoever) where I might hve
influence on that behaviour.
but thank you anyway for at least a reaction
bine

On Nov 13, 11:05 am, bine <sabine.hubrig-schaumb...@sungard.de> wrote:
> I have the confusing fact that in contrary to Linux (UnitedLinux or
> SuseEnterpriseServer)
> I miss some informations I write to a file.
> It seems that defining this file to be used for stdout+stderr in a csh-
> call like in
> myprog ... >>& mylog
> prevents later in the callhirarchy,
> that is by things called from myprog,
> and other scripts running in parallel as well
> (because this csh is running with nohup parallel to those)
> to put their output explicitly, e.g. by
> echo ... >> mylog
> into the same file.
>
> Is there some parameter or "set" command to make AIX behave like the
> others?
> I know that if those messages REALLY come in parallel, there MUST be
> some conflict
> due to the serialization of the output, so I might get things in a
> "wrong" order
> (I really don't want to discuss if this application is very
> sophisticated
> or should be modernized/recoded to a certain extent ;-),
> but I miss EVERYTHING of the other processes, not only some of the
> letters,
> so the question occurs if it's a configurable behavior of AIX?
>
> The highest level of any bos.*-module of the machine in lslpp-output
> is 5.3.0.61,
> but I think the behaviour hasn't changed and was the same (and always
> frustrating) in former releases?
>
> Thanks in advance
> bine


1 )Provide a stripped down version of your scripts which does
reproduce the problem
2) AFAIK is csh depreciated for shell programming. Go at least for the
tcsh and see if you have the same problem.
3) You are really sure that no process is writing with > to your
output file ?
4) I believe i have seen some problems with csh and pipes and grep
where the buffer was not flushed if the data was not getting larger
then 4k meaning the script output never left the buffer.

hth
Hajo

bine <sabine.hubrig-schaumb...@sungard.de> wrote:
> I miss some informations I write to a file.
> bsh <brian_hi...@rocketmail.com> wrote:
> > ...
> > I cannot tell by your description whether you do or do not
> > ultimately require _either_ serialization or just successful
> > writes to "mylog" of your "other scripts".

> I MISS a lot of output which brought me to the idea that on
> AIX there might be some locking-mechanism involved whereras
> on my other systems I get the lines.

I thought you would say that. All my experiences with past
AIX idiosyncrasies have been bad -- I try to avoid this OS as
much as possible, so I am skeptical that there is an easy fix.
However, I did some research and asked around, and here is what
I have been able to determine:

(1) I can find no (obvious) file-locking configuration
either in smit or smitty. Perhaps it's there, but no luck.

(2) Investigate "AIX Fast Connect"
$ net config /parameter_namearameter_value
configuration file: /etc/cifs/cifsConfig
investigate parameters: oplock_unix_lock, oplock_unix_lock_timeout,
oplockfiles, and oplocktimeout

(3) Investigate "Basic Networking Utilities (BNU)":
configuration directory: /etc/locks/

Are your processes long duration? OS-level file-locking could
be timing out. Also, is mylog on an NFS mount? AIX4 has a bug
that buffered I/O does not work properly with file locking.

"Understanding Locking":
http://publib.boulder.ibm.com/infoce...tc/locking.htm

Are you the system administrator of this system? Is there no
local expert to confer with? I recommend posting to the
newsgroup comp.unix.aix.

Perhaps the only practical recourse is to utilize an
external manager:

"dbappend.c": robust, serialized, recoverable appending to file
http://www.johncon.com/john/archive/dbappend.shar.gz
http://www.johncon.com/john/archive/dbappend.txt

=Brian

it may not be the silver bullet, but have you tried this yet ?

http://publib.boulder.ibm.com/infoce...erf_tuning.htm

Rgds
Mark Taylor

On 14 Nov., 19:44, Hajo Ehlers <serv...@metamodul.com> wrote:
> On Nov 13, 11:05 am, bine <sabine.hubrig-schaumb...@sungard.de> wrote:
>
>
>
>
>
> > I have the confusing fact that in contrary to Linux (UnitedLinux or
> > SuseEnterpriseServer)
> > I miss some informations I write to a file.
> > It seems that defining this file to be used for stdout+stderr in a csh-
> > call like in
> > myprog ... >>& mylog
> > prevents later in the callhirarchy,
> > that is by things called from myprog,
> > and other scripts running in parallel as well
> > (because this csh is running with nohup parallel to those)
> > to put their output explicitly, e.g. by
> > echo ... >> mylog
> > into the same file.
>
> > Is there some parameter or "set" command to make AIX behave like the
> > others?
> > I know that if those messages REALLY come in parallel, there MUST be
> > some conflict
> > due to the serialization of the output, so I might get things in a
> > "wrong" order
> > (I really don't want to discuss if this application is very
> > sophisticated
> > or should be modernized/recoded to a certain extent ;-),
> > but I miss EVERYTHING of the other processes, not only some of the
> > letters,
> > so the question occurs if it's a configurable behavior of AIX?
>
> > The highest level of any bos.*-module of the machine in lslpp-output
> > is 5.3.0.61,
> > but I think the behaviour hasn't changed and was the same (and always
> > frustrating) in former releases?
>
> > Thanks in advance
> > bine
>
> 1 )Provide a stripped down version of your scripts which does
> reproduce the problem
> 2) AFAIK is csh depreciated for shell programming. Go at least for the
> tcsh and see if you have the same problem.
> 3) You are really sure that no process is writing with > to your
> output file ?
> 4) I believe i have seen some problems with csh and pipes and grep
> where the buffer was not flushed if the data was not getting larger
> then 4k meaning the script output never left the buffer.
>
> hth
> Hajo- Zitierten Text ausblenden -
>
> - Zitierten Text anzeigen -

thanks to you guys, I will have a look into all those ideas ...
just some quick answers to some of your questions:

- AIX is not avoidable (due to our customers requests) as much as I'd
like ;-(

- I'm not the administrator so I can't try everything, but
unfortunately there's no other expert than me ;-) [or to more precise
";-(" ]

- mylog is not on NFS

- the csh starting the process using >>& is running "long" that is
normally some hours as it's kind of a demon to serve requests when
they are entered in the databasetables (and start script to do so,
which fail to put their (short) messages into the same mylog

- using tcsh would mean to change a lot of scripts which now start
with #!/bin/csh (because I couldn't rely on csh being linked to tcsh
on all [customer's] machines...) but of course it would be worth a try
for testing the effect only with my environment to eliminate that idea

- I'm sure that no process does >mylog because the start-messages of
the process the csh starts and others of that ARE in the file, only
those of other processes writing >>mylog fail to do so, so there's no
"over"writing of mylog but just a "non"writing to mylog

- hajo's 4th point is my favourite, that is: a buffer of 4K which
never gets

thank you guys so far
and if I detect something I'll post it

bine wrote:

> I have the confusing fact that in contrary to Linux (UnitedLinux or
> SuseEnterpriseServer)
> I miss some informations I write to a file.
> It seems that defining this file to be used for stdout+stderr in a csh-
> call like in
> myprog ... >>& mylog
> prevents later in the callhirarchy,
> that is by things called from myprog,
> and other scripts running in parallel as well
> (because this csh is running with nohup parallel to those)
> to put their output explicitly, e.g. by
> echo ... >> mylog
> into the same file.

One possibility is that csh on AIX might not set O_APPEND for
fd 1, it could just seek to the end before executing myprog.

In that case, using: sh -c 'myprog ... >> mylog 2>&1'
to start myprog would stop myprog (and things called from it,
assuming they write to stdout rather than reopening the log)
from overwriting the output from the other scripts. However,
the scripts, if also using csh, could still overwrite output
from myprog (and things called from it), and from each other.

--
Geoff Clare <netnews@gclare.org.uk>