SEO

I'd be GREATLY APRECIATED if someone could bail me out of this one! I
logged into my server an hour ago just to find that the root partition
is 100% filled:

/dev/mapper/VolGroup00-LogVol00
34285296 32544556 0 100% /

Because of that (I think), the data directory associated with my MySQL
installation got shot (read: disappeared!).

What would be the best way to recover the mysql data? The MySQL daemon
isn't running, and I can't find anything under lost+found, either!
Please help for I *really* need to get [some or most of] the data back!

On 2006-03-08, im2kul2@gmail.com <im2kul2@gmail.com> wrote:
> logged into my server an hour ago just to find that the root partition
> is 100% filled:
>
> Because of that (I think), the data directory associated with my MySQL
> installation got shot (read: disappeared!).

My experience is that directories don't disappear "automagically", someone
zapped it or the disk is b0rken.

> What would be the best way to recover the mysql data?

Restore from a backup. You *do* have backups, right?

Davide

--
Why would people waste their time developing viruses for Microsoft
products when Microsoft does such a good job itself of adding in bugs
which crash your system?
-- From a Slashdot.org post

Davide Bianchi <davideyeahsure@onlyforfun.net> wrote:
>> logged into my server an hour ago just to find that the root partition
>> is 100% filled:
>>
>> Because of that (I think), the data directory associated with my MySQL
>> installation got shot (read: disappeared!).
>
> My experience is that directories don't disappear "automagically", someone
> zapped it or the disk is b0rken.
>
>> What would be the best way to recover the mysql data?
>
> Restore from a backup. You *do* have backups, right?
>
> Davide

Davide? Sounds more like Peter....

Questions to the original poster:

- Is the MySQL data directory in the root partition or on another
filesystem?

- Did you run a file system check before you looked in lost+found? Only
a filesystem check puts data there.

Things for you to do:

- Boot the system from a rescue CD and check the root file system.

- Verify that the MySQL data reside on the root partition (check with
/etc/fstab).

- If all looks good, mount the root partition read-write and delete some
files.

- If things do not look good, do not mount the partition read-write.
If necessary, contract a professional who can help you salvage the data.

Things to do next time to avoid the problem:

- Do backups.

- Do not keep application data on the root partition.

Yours,
Laurenz Albe

Laurenz Albe wrote:
> Davide Bianchi <davideyeahsure@onlyforfun.net> wrote:
>>> logged into my server an hour ago just to find that the root
>>> partition is 100% filled:
>>>
>>> Because of that (I think), the data directory associated with my
>>> MySQL installation got shot (read: disappeared!).
>>
>> My experience is that directories don't disappear "automagically",
>> someone zapped it or the disk is b0rken.
>>
>>> What would be the best way to recover the mysql data?
>>
>> Restore from a backup. You *do* have backups, right?
>>
>> Davide
>
> Davide? Sounds more like Peter....

For newbies, Laurenz is referring to Peter Breuer. I haven't seen him here
lately.

> Questions to the original poster:
>
> - Is the MySQL data directory in the root partition or on another
> filesystem?
>
> - Did you run a file system check before you looked in lost+found?
> Only a filesystem check puts data there.
>
> Things for you to do:
>
> - Boot the system from a rescue CD and check the root file system.
>
> - Verify that the MySQL data reside on the root partition (check with
> /etc/fstab).
>
> - If all looks good, mount the root partition read-write and delete
> some files.

And look for what overflowed. /var/log is a fun place to have overflow if
you have immense log files that never get rotated, which is easy to do
accidentally. /usr/local is another if you're dropping in big locally
compiled software bundles, as is /tmp when downloading things like ISO
images with some web browsers. I find it very difficult to predict what
partition will next need a few gig of space, so really despise making lots
and lots of partitions for /, /usr, /tmp, /var, /daata, /opt, etc., etc.,
etc.

> - If things do not look good, do not mount the partition read-write.
> If necessary, contract a professional who can help you salvage the
> data.
>
> Things to do next time to avoid the problem:
>
> - Do backups.
>
> - Do not keep application data on the root partition.

And keep simple, *LARGE* spaces in your partitions available.

And monitor your systems for overflowing disks with some way to report it: I
rather like Nagios for that, myself.

im2kul2@gmail.com wrote:
> I'd be GREATLY APRECIATED if someone could bail me out of this one! I
> logged into my server an hour ago just to find that the root partition
> is 100% filled:
>
> /dev/mapper/VolGroup00-LogVol00
> 34285296 32544556 0 100% /
>
> Because of that (I think), the data directory associated with my MySQL
> installation got shot (read: disappeared!).
>
> What would be the best way to recover the mysql data? The MySQL daemon
> isn't running, and I can't find anything under lost+found, either!
> Please help for I *really* need to get [some or most of] the data back!

Log in as root and ls to see what is filling it with du. I've had it happen on
a small drive and a cron task always failing (mrtg?) and it kept sending me
email about it failing. From that I learned to check root mail at least once a
year.

--
It is more accurate to compare Israel to the communists than the Nazis
because they admit it and the communists were worse.
-- The Iron Webmaster, 3578
nizkor http://www.giwersworld.org/nizkook/nizkook.phtml
environmentalism http://www.giwersworld.org/environment/aehb.phtml a9

Matt Giwer wrote:
> im2kul2@gmail.com wrote:
>> I'd be GREATLY APRECIATED if someone could bail me out of this one! I
>> logged into my server an hour ago just to find that the root partition
>> is 100% filled:
>>
>> /dev/mapper/VolGroup00-LogVol00
>> 34285296 32544556 0 100% /
>>
>> Because of that (I think), the data directory associated with my MySQL
>> installation got shot (read: disappeared!).
>>
>> What would be the best way to recover the mysql data? The MySQL daemon
>> isn't running, and I can't find anything under lost+found, either!
>> Please help for I *really* need to get [some or most of] the data back!
>
> Log in as root and ls to see what is filling it with du. I've had it
> happen on a small drive and a cron task always failing (mrtg?) and it
> kept sending me email about it failing. From that I learned to check
> root mail at least once a year.
>
Indeed.

After many years of unattended Unix machines we learned to make root (/)
small, because it never changes, /var separate and very large, as if
anything goes mad thats where trash ends up, and install a /home for
user data.

When using an SQL app, that either goes in /home. or in its own /mysql
partition.

Yes, things need adapting for this file layout, but it paid dividends.

And if using a transaction file, that goes on a different DISK to the
main sql data.

That way, with nightly backups of both, you can always restore
accurately if a disk goes down.

Please use "find / -size +100000000C|page" to find out any files bigger
than 100MB and delete those junk files only if you can figure them out.
Good luck. Thomas

Thomas wrote:
> Please use "find / -size +100000000C|page" to find out any files bigger
> than 100MB and delete those junk files only if you can figure them out.
> Good luck. Thomas
>
I think some combination of smaller size, mtime (limit to recently
written files), and staying in the filesystem of interest.

example:
find / -xdev -type f -mtime -2 -size +2000000

Then let's sort by size and save in a file...
find / -xdev -type f -mtime -2 -size +2000000 -print0 |
xargs -0 ls -s | sort -n | tee file_in_another_filesystem | tail -20

That will show the largest of the large files modified in the last two
days, the "print0" handles filenames with special characters.

Other tricks include just using the "-ls" in find and "sort +6n" to
order on size. Whatever works for you.

--
bill davidsen
SBC/Prodigy Yorktown Heights NY data center
http://newsgroups.news.prodigy.com