SEO

Hey all,
I am experiencing some very strange problems on two boxes running
Slackware 9.0. Here are some of the details:

System 1: generic AMD Athlon 900, Tekram SCSI card, IBM 36GB SCSI disk
System 2: generic Intel 600Mhz PIII, PIIX IDE, 16GB IDE disk

Both boxes previously ran other, older versions of Slackware on the
same hardware in the past w/o problems.

Both systems are newly reinstalled with Slack 9. Default kernel of
2.4.20, custom compiled and extremely stripped. I do not use LKMs or
compile the kernel with module support. All file systems run
ReiserFS.

System 1 runs normal "net" services--httpd, sendmail, POP3, IMAP, ssh.
It has local users.
System 2 is very stripped down. No users capable of local access
other than root. It runs sshd, syslog -r (its a log repository for
some routers), MRTG, big brother (neither of which open ports), httpd.

System 1 was installed on August 27th and has been up since then. It
has run fine until early morning of Sept 2nd.

System 2 was installed on August 26th and "went bad" (I'll explain
more later) on August 29th. I reinstalled on the 30th and it almost
immediately went bad on me while I was on the console doing
post-configuration. At the time the only thing running was sshd and
possibly httpd.

Now, what has been happening:

Various system binaries, mostly in /bin are changing in size and
thusly seg faulting and failing to operate. Nearly every binary in
/bin is affected, including mount, umount, sync, rm, gzip, cp, more,
etc. Oddly, ps has the correct file size (I am comparing with a known
good slack 9 system). A few binaries in /sbin are affected as well.
I haven't found anything in /lib, or /usr/lib that is different. I
just noticed that binaries in /usr/local/bin, all of which are custom
installed binaries, are affected as well.

Comparing the output of strings on sample binaries on an 'affected'
system versus a clean system shows that the binaries are exactly the
same for most of the file and then the bad binary begins to show ASCII
gibberish in the strings output whereas the clean file does not.

Now my first thought was immediately "hack!", but I don't think this
is the case here. I've seen many hacks of many linux boxes, and while
I am not an expert, I have done enough forensics of hacked linux boxes
to get 'gut feelings' of whether a box has been rooted.

Here are a few reasons why I don't think the boxes have been rooted:

1. I don't see a need for a hacker or rootkit to replace binaries such
as cp, more, gzip, sync, etc. Usually things like ps, ls, find,
netstat, etc., are replaced. Often an LKM is installed to further
obfuscate things, but that is not possible on these two boxes.

2. While writing this post and investigating /usr/local/bin, I noticed
that custom compiled and installed software like pngtogd and gd2topng
(part of the GD graphics library) are also affected. There is no
reason for a hacker to touch files like this.

3. System #2 "went bad" while I was on the console in the process of
configuring it. The only user capable of logging into that box is
root and I would have noticed console messages had root logged in
remotely via SSH while I was working on the system. Also, it takes at
least a little bit of time for a script-kiddie to compile their
software, clean out log files, etc., during which time they would be
naked and visible.

4. For the most part, script-kiddies and amateur hackers are extremely
careless and leave signs of their activities all over the system.
(.history files, stuff in /tmp, not noticing that my syslog.conf is
not the default and I log to different file names, etc.). Neither
system shows any evidence at all of activity, other than the
differently sized binaries.

Oddities that point to a possible hack:

1. Both boxes show the exact same file sizes on affected files. The
files are always the same ones being affected. Could be indicative of
a standard root kit, but again, why would a root kit replace gd
executables?

Something that just occurred to me is that I believe the file system
corruption occurred on each box right after I scp'ed a file from a
different system to that box.

On system #2, before it went bad, I was having a problem with
in.tftpd. After doing a little research I found that in.tftpd is
broken in slack 8.1 and 9.0. People reported success with versions
from 8.0 and earlier. I scp'ed a copy of in.tftpd from an older box.
Shortly thereafter the problems began and I had to rebuild the system.

On system #1, a user needed a copy of swish-e, which I had on another
system, so I scp'ed it to the system the night of Sept 1st. The
system 'went bad' on Sept. 2nd.

The above may be just coincidental though.

Does anyone have any thoughts on a very puzzling and frustrating
problem?

Thanks in advance for any ideas or suggestions!
Jay

In article <ec4082b3.0309012255.24f2e7c8@posting.google.com >, Aleksandr wrote:
>
> Now my first thought was immediately "hack!"

My first thought was "bad memory". Your symptoms seem indicative of
that, except for the part about the files becoming the same size on
each of the two boxes. Can you verify that the files are the same using
md5sum? If the files are the same, then it almost definately is not bad
memory! Having two boxes with different memory chips appending the same
random data to the same files is just too weird.

> 2. While writing this post and investigating /usr/local/bin, I noticed
> that custom compiled and installed software like pngtogd and gd2topng
> (part of the GD graphics library) are also affected. There is no
> reason for a hacker to touch files like this.

Hmm... I've never heard of anyone with your condition before, but you
are using some custom compiled software which I'm not familiar with. Is
this software common to both machines, and if so, could it be the
source of your trouble?

> 1. Both boxes show the exact same file sizes on affected files. The
> files are always the same ones being affected. Could be indicative of
> a standard root kit, but again, why would a root kit replace gd
> executables?

Again, verify that these files are identical with md5sum.

> Something that just occurred to me is that I believe the file system
> corruption occurred on each box right after I scp'ed a file from a
> different system to that box.

That doesn't make a whole lot of sense though. scp is just a simple
copy program tunneled through ssh. Why would that touch the majority of
your files in /bin? Unless possibly you scp'ed these files from the
same machine, and that machine has something very wrong with it.

> The above may be just coincidental though.

Most likely they are, but that's just too damned coincidental not to
mean something if you ask me (and you did).

> Thanks in advance for any ideas or suggestions!

Let us know when you get it fixed. I'm going to try to scp something
from another machine to a slackware 9.0 box today and see what happens.

--
It is better to hear the rebuke of the wise,
Than for a man to hear the song of fools.
Ecclesiastes 7:5

Alan Hicks wrote:
> In article <ec4082b3.0309012255.24f2e7c8@posting.google.com >, Aleksandr wrote:
>> 1. Both boxes show the exact same file sizes on affected files. The
>> files are always the same ones being affected. Could be indicative of
>> a standard root kit, but again, why would a root kit replace gd
>> executables?
>
> Again, verify that these files are identical with md5sum.

i cannot imagine anything but a cracker if the files are indeed
identical. perhaps someone is just messing up your boxen to watch you
squirm..

>> Something that just occurred to me is that I believe the file system
>> corruption occurred on each box right after I scp'ed a file from a
>> different system to that box.
>
> That doesn't make a whole lot of sense though. scp is just a simple
> copy program tunneled through ssh. Why would that touch the majority of
> your files in /bin? Unless possibly you scp'ed these files from the
> same machine, and that machine has something very wrong with it.

or scp (or ssh?) is compromised?

i don't really know, though. never had to deal with such issues.

--
Joost Kremers
since when is vi an editor? a discussion on vi belongs in
comp.tools.unusable or something...

Joost Kremers wrote:
> i cannot imagine anything but a cracker if the files are indeed
> identical.

Neither can I; it's just very strange not to see any other signs.
That's why it would be helpful to do an md5sum to verify they are
indeed indentical. It could be that he used something like 'ls -lh' to
list the files and got approximations.

>> That doesn't make a whole lot of sense though. scp is just a simple
>> copy program tunneled through ssh. Why would that touch the majority of
>> your files in /bin? Unless possibly you scp'ed these files from the
>> same machine, and that machine has something very wrong with it.
>
> or scp (or ssh?) is compromised?

That would be my conclusion. It seems likely even that this could be
the case if he is scp;ing these files from the same machine. You've got
to look at what is common between these two boxes to see what is
causing the behavior, and it seems scp or ssh is our culprit. By any
chance, you wouldn't be using the same root password on all these boxes
now, would you?

Alan Hicks <someone@news4.sucknews.com> wrote in message news:<slrnbl9f0q.11p.someone@sentinel.custom-consulting.com>...
> Joost Kremers wrote:
> > i cannot imagine anything but a cracker if the files are indeed
> > identical.
>
> Neither can I; it's just very strange not to see any other signs.
> That's why it would be helpful to do an md5sum to verify they are
> indeed indentical. It could be that he used something like 'ls -lh' to
> list the files and got approximations.
>
> >> That doesn't make a whole lot of sense though. scp is just a simple
> >> copy program tunneled through ssh. Why would that touch the majority of
> >> your files in /bin? Unless possibly you scp'ed these files from the
> >> same machine, and that machine has something very wrong with it.
> >
> > or scp (or ssh?) is compromised?
>
> That would be my conclusion. It seems likely even that this could be
> the case if he is scp;ing these files from the same machine. You've got
> to look at what is common between these two boxes to see what is
> causing the behavior, and it seems scp or ssh is our culprit. By any
> chance, you wouldn't be using the same root password on all these boxes
> now, would you?

I've got a few updates from further research I did today.

I reinstalled system #2 from scratch again today, and sure enough it
"went bad" on me about 15 minutes after the reinstall. Based on what
I was doing at the time, I now highly doubt the possibility of
hacking. The only service running at the time was sshd, root was the
only user on the system, and the password had been randomly created 10
minutes earlier (8 characters, mixed case, totally random, etc.).

In order for this box to have been hacked, someone would have had to
have a remote exploit for OpenSSH 3.6.1, which I don't believe exists,
or brute-forced a random password in about 10 minutes, all while I was
on the system console. Highly unlikely set of circumstances.

So, now I am looking at the possibility that my install media may be
bad. As this is a "custom" slackware 9, this may be possible.
Basically, I took a stock slack 9 CD1, updated any security issues
with new packages from slackware.com, created custom tagfiles, and
added extra local software. An ISO was created of this and then
burned to CD.

The only thing I have against the possibility of bad install media is
that at least one of my 'good' slack 9 systems was installed from the
exact same CD and has been up and stable for more than 60 days now.
All of the systems that have become corrupted have done so within a
time frame of 10 minutes to 3 or 4 days.

This got me into thinking deeply about what may be different about the
server that is still functioning, and the ones that keep getting
corrupted. Aside from hardware differences, the only difference I can
find is that the good server is running Apache 2.0.x, while the bad
servers are running Apache 1.3.27. These versions of apache are all
custom-compiled add-on software, not the copy included in Slack 9.

I find it hard to believe that Apache could cause problems like this,
but so far this whole issue has been hard to believe. I will do a bit
more testing to try to narrow down the problem further.

Aleks

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, Sep 02, 2003 at 05:17:18PM -0700, Aleksandr wrote:

> I reinstalled system #2 from scratch again today, and sure enough it
> "went bad" on me about 15 minutes after the reinstall. Based on what

try installing one, without networking it. see if it goes bad on you
like that. put all extra sources you like to compile on a cd, and use
that, if you need extra things not included on your slackware CD.

verify the source tarballs with the original files on the ftp site,
making sure signatures match. you'ge got some crazy things happening
there


Jurgen.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE/VZ5z1ucXIiwNwbURArgkAJ0cfWQ5LVkzGGDW18GpfmVnMPTX5A CfeRj7
3vy/tp0AaZy9jLJT1AmLE6M=
=EUih
-----END PGP SIGNATURE-----

Jurgen Philippaerts <jurgen@see.my.pgp.key> wrote in message news:<20030903075533.GA365@anubis.is.traumatized.o rg>...
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> verify the source tarballs with the original files on the ftp site,
> making sure signatures match. you'ge got some crazy things happening
> there
>

Well, I think I have finally narrowed down and eliminated the problem.
It seems to have been caused by a particular install CD I was using.
I have two copies of my custom slack 9 CD. Both are identical (or
should be), other than the kernel used for booting. The "bad" CD uses
the stock kernels included with the Slack 9 CD1. On the good CD I
recompiled all of the kernels to use SMP and disable USB because of a
weird conflict I was having with an Adaptec SCSI card (it only works
with an SMP kernel).

I finally realized that the Slack 9 machines that were not getting
corrupted had all been installed with the SMP kernel CD, while the two
systems I was writing about here had been constantly installed and
reinstalled with the stock kernel CD. Perhaps something got munged
during the ISO creation or the CD burning, or even afterwards. At
this point, I don't even want to bother to figure out what was wrong
with it. I cracked the bad CD in half in front of my other CDs as a
lesson to them

Thanks everyone for all of the suggestions put forth!

Aleks

Aleksandr wrote:
> with it. I cracked the bad CD in half in front of my other CDs as a
> lesson to them

well, that should teach them! ;-)

--
Joost Kremers
since when is vi an editor? a discussion on vi belongs in
comp.tools.unusable or something...