Hey all,
I am experiencing some very strange problems on two boxes running
Slackware 9.0. Here are some of the details:

System 1: generic AMD Athlon 900, Tekram SCSI card, IBM 36GB SCSI disk
System 2: generic Intel 600Mhz PIII, PIIX IDE, 16GB IDE disk

Both boxes previously ran other, older versions of Slackware on the
same hardware in the past w/o problems.

Both systems are newly reinstalled with Slack 9. Default kernel of
2.4.20, custom compiled and extremely stripped. I do not use LKMs or
compile the kernel with module support. All file systems run
ReiserFS.

System 1 runs normal "net" services--httpd, sendmail, POP3, IMAP, ssh.
It has local users.
System 2 is very stripped down. No users capable of local access
other than root. It runs sshd, syslog -r (its a log repository for
some routers), MRTG, big brother (neither of which open ports), httpd.

System 1 was installed on August 27th and has been up since then. It
has run fine until early morning of Sept 2nd.

System 2 was installed on August 26th and "went bad" (I'll explain
more later) on August 29th. I reinstalled on the 30th and it almost
immediately went bad on me while I was on the console doing
post-configuration. At the time the only thing running was sshd and
possibly httpd.

Now, what has been happening:

Various system binaries, mostly in /bin are changing in size and
thusly seg faulting and failing to operate. Nearly every binary in
/bin is affected, including mount, umount, sync, rm, gzip, cp, more,
etc. Oddly, ps has the correct file size (I am comparing with a known
good slack 9 system). A few binaries in /sbin are affected as well.
I haven't found anything in /lib, or /usr/lib that is different. I
just noticed that binaries in /usr/local/bin, all of which are custom
installed binaries, are affected as well.

Comparing the output of strings on sample binaries on an 'affected'
system versus a clean system shows that the binaries are exactly the
same for most of the file and then the bad binary begins to show ASCII
gibberish in the strings output whereas the clean file does not.

Now my first thought was immediately "hack!", but I don't think this
is the case here. I've seen many hacks of many linux boxes, and while
I am not an expert, I have done enough forensics of hacked linux boxes
to get 'gut feelings' of whether a box has been rooted.

Here are a few reasons why I don't think the boxes have been rooted:

1. I don't see a need for a hacker or rootkit to replace binaries such
as cp, more, gzip, sync, etc. Usually things like ps, ls, find,
netstat, etc., are replaced. Often an LKM is installed to further
obfuscate things, but that is not possible on these two boxes.

2. While writing this post and investigating /usr/local/bin, I noticed
that custom compiled and installed software like pngtogd and gd2topng
(part of the GD graphics library) are also affected. There is no
reason for a hacker to touch files like this.

3. System #2 "went bad" while I was on the console in the process of
configuring it. The only user capable of logging into that box is
root and I would have noticed console messages had root logged in
remotely via SSH while I was working on the system. Also, it takes at
least a little bit of time for a script-kiddie to compile their
software, clean out log files, etc., during which time they would be
naked and visible.

4. For the most part, script-kiddies and amateur hackers are extremely
careless and leave signs of their activities all over the system.
(.history files, stuff in /tmp, not noticing that my syslog.conf is
not the default and I log to different file names, etc.). Neither
system shows any evidence at all of activity, other than the
differently sized binaries.

Oddities that point to a possible hack:

1. Both boxes show the exact same file sizes on affected files. The
files are always the same ones being affected. Could be indicative of
a standard root kit, but again, why would a root kit replace gd
executables?

Something that just occurred to me is that I believe the file system
corruption occurred on each box right after I scp'ed a file from a
different system to that box.

On system #2, before it went bad, I was having a problem with
in.tftpd. After doing a little research I found that in.tftpd is
broken in slack 8.1 and 9.0. People reported success with versions
from 8.0 and earlier. I scp'ed a copy of in.tftpd from an older box.
Shortly thereafter the problems began and I had to rebuild the system.

On system #1, a user needed a copy of swish-e, which I had on another
system, so I scp'ed it to the system the night of Sept 1st. The
system 'went bad' on Sept. 2nd.

The above may be just coincidental though.

Does anyone have any thoughts on a very puzzling and frustrating
problem?

Thanks in advance for any ideas or suggestions!
Jay