Steven Mocking <s.mocking@gmail.com> wrote:
> Over ssh, Ctrl-C and Ctrl-4 couldn't recover the prompt. We managed to
> gain the ability to execute commands on it remotely as root by a not
> entirely intentional backdoor, so we could investigate and fix the
> machine. After a while we managed to get a process list. There were a
> few dozen defunct sshd children and smb also had children. The most
> striking was a whole bunch of crond children which seemed to have hung
> up.

Well, what are they waiting on? Run ps axl and check the WCHAN column.


> root 1177 0.0 0.0 1568 148 ? S Nov07 0:02 crond
> root 1082 0.0 0.0 1568 108 ? S 04:42 0:00 \_
> CROND
> smmsp 1084 0.0 0.0 5760 684 ? S 04:42 0:00 | \_
> [sendmail]
> root 1091 0.0 0.0 1564 64 ? D 05:01 0:00 \_
> CROND
> root 1093 0.0 0.0 1564 64 ? D 05:01 0:00 \_
> CROND
> root 1095 0.0 0.0 1564 64 ? D 05:01 0:00 \_
> CROND
> root 1109 0.0 0.0 1564 64 ? D 06:01 0:00 \_
> CROND
> root 1111 0.0 0.0 1564 64 ? D 06:01 0:00 \_
> CROND
> root 1113 0.0 0.0 1564 64 ? D 06:01 0:00 \_
> CROND
> The list continued like this until the present - three hanging children
> every hour. Syslogd was also still running, even though

Shrug - you have some hardware problem. Probably a fubared disk. Maybe
just a NFS mount that has gone bad. Find out.

> /var/log/messages was empty:
> root 1037 0.0 0.0 1464 564 ? S Nov07 0:11 syslogd -m 0

??? What do you mean?


> Strangely, the /var/log/messages file seemed empty. We then attempted
> to restart sshd to see if that would allow logins. This got stuck and

Well, where did it stick? Strace it.

> But now, /var/log/messages was no longer empty - there was a rather
> conflicting and suspicious line at the top (note the timestamp).

> Dec 1 14:42:45 tk7 syslogd 1.4.1: restart.
> Dec 1 04:08:03 tk7 su(pam_unix)[1026]: session opened for user news by (uid=0)
> Dec 1 04:08:03 tk7 su(pam_unix)[1026]: session closed for user news
> Dec 1 04:42:00 tk7 CROND[1078]: (root) CMD (run-parts /etc/cron.monthly)
> Dec 1 04:42:00 tk7 CROND[1083]: (root) CMD (root run-parts /etc/cron.monthly)

I see nothing suspicious. Time adjust. So? Reset the bios clock so the
adjust won't be necessary.

> /var/log/boot.log also seems to have moved back in time:

> Dec 1 15:45:47 tk7 sshd:
> Dec 1 15:45:47 tk7 sshd: succeeded
> Dec 1 15:45:48 tk7 tomcat: Starting tomcat:
> Dec 1 15:45:48 tk7 tomcat: tomcat.
> Dec 1 15:45:49 tk7 tomcat: Usage: /etc/init.d/tomcat {start|stop|restart|force-reload}
> Dec 1 15:45:49 tk7 rc: Starting tomcat: failed
> Dec 1 14:48:24 tk7 ntpd: succeeded
> Dec 1 14:48:24 tk7 ntpd: ntpd startup succeeded
> Dec 1 14:48:26 tk7 nfs: Starting NFS services: succeeded
> Dec 1 14:48:26 tk7 nfs: rpc.rquotad startup succeeded
> Dec 1 14:48:27 tk7 nfs: rpc.nfsd startup succeeded
> Dec 1 14:48:27 tk7 nfs: rpc.mountd startup succeeded

Well, you certainly have whacky clocking.

> Does this sound familiar to someone?

You forgot to trace what was happening. There isn't any data heer to
point anywhere.

Peter