Bill Marcum wrote:
> On 28 Jul 2006 11:28:54 -0700, jplee3@gmail.com
> <jplee3@gmail.com> wrote:
> >
> > Thanks for all the help.
> > Well, the other big problem we're having is that the server keeps going
> > down - it's gone down like 4 times in the past week. It usually goes
> > down around 4am, the same time that the cron.daily (including the
> > logrotate script), runs. One of my coworkers and I have a hunch that
> > something is conflicting there - perhaps with another script that's
> > trying to run:
> >
> > 0anacron logrotate prelink tmpwatch
> > 0logwatch makewhatis.cron rpm yum.cron
> > cups mlocate.cron tetex.cron
> >
> Have you tried running top around 4 am? It could be mlocate building an
> index of those exponentially multiplying log files. Or maybe your
> server gets a lot of clients at that time, but you probably would have
> seen that in the server logs.
>
>
>
> --
> In just seven days, I can make you a man!
> -- The Rocky Horror Picture Show

Ok guys, I think we figured the server-crash portion out. The sysadmin
commented out the /wtmp entry and instead put in his own entry:

"create 0755 root utmp" - basically, it looks like this overwrites
utmp, which belongs to /wtmp - I'm not sure why, in detail, this would
cause everything to crash, but the server's been up ever since we
commented out that entire entry


And regarding the exponential growth of numbers (thanks unruh!):

"Anyway, someone stupidly put something like
/var/log/syslog* into their logrotate rules. That means rotate syslog,
syslog.1, syslog.1.1"

- I believe I've found the cause. The sysadmin edited the syslog script
in the same manner (his intention was to use the asterisk to grab all
files in the nt directory; of course, it didn't turn out to work that
way):

/var/log/nt/* /var/log/messages /var/log/secure /var/log/maillog
/var/log/spoole
r /var/log/boot.log /var/log/cron {
sharedscripts
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2>
/dev/null || t
rue
endscript
}
/etc/logrotate.d/syslog

jplee3@gmail.com writes:


....
>And regarding the exponential growth of numbers (thanks unruh!):

>"Anyway, someone stupidly put something like
>/var/log/syslog* into their logrotate rules. That means rotate syslog,
>syslog.1, syslog.1.1"

>- I believe I've found the cause. The sysadmin edited the syslog script
>in the same manner (his intention was to use the asterisk to grab all
>files in the nt directory; of course, it didn't turn out to work that
>way):

Actually it worked exactly that way, except he did not want it to work that
way. Ie, instead of all files he really wanted only the ones without a
..number at the end of them. Very very easy mistake to make (even the
experts at Mandrake made it), and once you make it once it is forever
burned in your memory.


>/var/log/nt/* /var/log/messages /var/log/secure /var/log/maillog
>/var/log/spoole
>r /var/log/boot.log /var/log/cron {
> sharedscripts
> postrotate
> /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2>
>/dev/null || t
>rue
> endscript
>}
>/etc/logrotate.d/syslog

jplee3@gmail.com wrote:

> Ok guys, I think we figured the server-crash portion out. The sysadmin
> commented out the /wtmp entry and instead put in his own entry:
>
> "create 0755 root utmp" - basically, it looks like this overwrites
> utmp, which belongs to /wtmp - I'm not sure why, in detail, this would
> cause everything to crash, but the server's been up ever since we
> commented out that entire entry

Something tells me that sysadmin's going to be buying everyone dinner.
The bit about setting utmp to 755 permissions, in particular, is very,
very bad. You want to leave utmp severely alone.

Rick Moen wrote:
> jplee3@gmail.com wrote:
>
> > Ok guys, I think we figured the server-crash portion out. The sysadmin
> > commented out the /wtmp entry and instead put in his own entry:
> >
> > "create 0755 root utmp" - basically, it looks like this overwrites
> > utmp, which belongs to /wtmp - I'm not sure why, in detail, this would
> > cause everything to crash, but the server's been up ever since we
> > commented out that entire entry
>
> Something tells me that sysadmin's going to be buying everyone dinner.
> The bit about setting utmp to 755 permissions, in particular, is very,
> very bad. You want to leave utmp severely alone.

Hahaha, well the dinner-part hasn't happened yet. And I don't suspect
it ever will