Hi Everybody,

According to section 23.3.2 of 8.2.4 manual:

Also, some versions of GNU tar consider it
an error if a file is changed while tar is
copying it. There does not seem to be any
very convenient way to distinguish this
error from other types of errors, other
than manual inspection of tar's messages.
GNU tar is therefore not the best tool for
making base backups.

On my linux machine, gnu tar is the tar. Does
anybody have a suggestion as to where I can go
to get a tar that is not gnu?

Thanks in advance.

Tena Sakai
tsakai@gallo.ucsf.edu

>>> On Tue, Aug 21, 2007 at 7:28 PM, in message
<FE44E0D7EAD2ED4BB2165071DB8E328C03062B59@egcrc-ex01.egcrc.org>, "Tena Sakai"
<tsakai@gallo.ucsf.edu> wrote:
>
> On my linux machine, gnu tar is the tar. Does
> anybody have a suggestion as to where I can go
> to get a tar that is not gnu?

Have you considered using cpio instead?

http://www.gnu.org/software/cpio/

-Kevin




---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

Tena Sakai wrote:
> Hi Everybody,
>
> According to section 23.3.2 of 8.2.4 manual:
>
> Also, some versions of GNU tar consider it
> an error if a file is changed while tar is
> copying it. There does not seem to be any
> very convenient way to distinguish this
> error from other types of errors, other
> than manual inspection of tar's messages.
> GNU tar is therefore not the best tool for
> making base backups.
>
> On my linux machine, gnu tar is the tar. Does
> anybody have a suggestion as to where I can go
> to get a tar that is not gnu?

We have updated the 8.3 documentation to be more accurate about GNU tar:

Some backup tools that you might wish to use emit warnings or errors
if the files they are trying to copy change while the copy proceeds.
This situation is normal, and not an error, when taking a base backup
of an active database; so you need to ensure that you can distinguish
complaints of this sort from real errors. For example, some versions
of <application>rsync</> return a separate exit code for
<quote>vanished source files</>, and you can write a driver script to
accept this exit code as a non-error case. Also, some versions of
GNU <application>tar</> consider it an error if a file is changed
while <application>tar</> is copying it. Fortunately, GNU
<application>tar</> versions 1.16 and later exit with <literal>1</>
if files changed during the backup, and <literal>2</> for other errors.

so your version of 'tar' might be fine.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

Hi Kevin,

Yes, I have, but I am much more familiar with tar.
I think I will go with the latest gnu tar (v 1.18)
which is suggested by Bruce. I will play with it
tomorrow and see how it goes.

Thanks.

Tena

tsakai@gallo.ucsf.edu


-----Original Message-----
From: Kevin Grittner [mailto:Kevin.Grittner@wicourts.gov]
Sent: Tue 8/21/2007 6:02 PM
To: Tena Sakai; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] tar, but not gnu tar

>>> On Tue, Aug 21, 2007 at 7:28 PM, in message
<FE44E0D7EAD2ED4BB2165071DB8E328C03062B59@egcrc-ex01.egcrc.org>, "Tena Sakai"
<tsakai@gallo.ucsf.edu> wrote:
>
> On my linux machine, gnu tar is the tar. Does
> anybody have a suggestion as to where I can go
> to get a tar that is not gnu?

Have you considered using cpio instead?

http://www.gnu.org/software/cpio/

-Kevin

Tena,

We have been very happy with star. It is a very nice pax,
cpio, gnutar,... replacement. You may want to give it a try.

Ken

On Tue, Aug 21, 2007 at 06:53:53PM -0700, Tena Sakai wrote:
> Hi Kevin,
>
> Yes, I have, but I am much more familiar with tar.
> I think I will go with the latest gnu tar (v 1.18)
> which is suggested by Bruce. I will play with it
> tomorrow and see how it goes.
>
> Thanks.
>
> Tena
>
> tsakai@gallo.ucsf.edu
>
>
> -----Original Message-----
> From: Kevin Grittner [mailto:Kevin.Grittner@wicourts.gov]
> Sent: Tue 8/21/2007 6:02 PM
> To: Tena Sakai; pgsql-admin@postgresql.org
> Subject: Re: [ADMIN] tar, but not gnu tar
>
> >>> On Tue, Aug 21, 2007 at 7:28 PM, in message
> <FE44E0D7EAD2ED4BB2165071DB8E328C03062B59@egcrc-ex01.egcrc.org>, "Tena Sakai"
> <tsakai@gallo.ucsf.edu> wrote:
> >
> > On my linux machine, gnu tar is the tar. Does
> > anybody have a suggestion as to where I can go
> > to get a tar that is not gnu?
>
> Have you considered using cpio instead?
>
> http://www.gnu.org/software/cpio/
>
> -Kevin
>
>
>
>

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

Thanks, Ken.

I just glanced at man page for star. It looks promising
and I will experiment with it. This may be the ticket.

Regards,

Tena
tsakai@gallo.ucsf.edu


-----Original Message-----
From: Kenneth Marshall [mailto:ktm@rice.edu]
Sent: Wed 8/22/2007 5:30 AM
To: Tena Sakai
Cc: Kevin Grittner; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] tar, but not gnu tar

Tena,

We have been very happy with star. It is a very nice pax,
cpio, gnutar,... replacement. You may want to give it a try.

Ken

On Tue, Aug 21, 2007 at 06:53:53PM -0700, Tena Sakai wrote:
> Hi Kevin,
>
> Yes, I have, but I am much more familiar with tar.
> I think I will go with the latest gnu tar (v 1.18)
> which is suggested by Bruce. I will play with it
> tomorrow and see how it goes.
>
> Thanks.
>
> Tena
>
> tsakai@gallo.ucsf.edu
>
>
> -----Original Message-----
> From: Kevin Grittner [mailto:Kevin.Grittner@wicourts.gov]
> Sent: Tue 8/21/2007 6:02 PM
> To: Tena Sakai; pgsql-admin@postgresql.org
> Subject: Re: [ADMIN] tar, but not gnu tar
>
> >>> On Tue, Aug 21, 2007 at 7:28 PM, in message
> <FE44E0D7EAD2ED4BB2165071DB8E328C03062B59@egcrc-ex01.egcrc.org>, "Tena Sakai"
> <tsakai@gallo.ucsf.edu> wrote:
> >
> > On my linux machine, gnu tar is the tar. Does
> > anybody have a suggestion as to where I can go
> > to get a tar that is not gnu?
>
> Have you considered using cpio instead?
>
> http://www.gnu.org/software/cpio/
>
> -Kevin
>
>
>
>

Hi Everybody,

I had a bit of time to experiment with tar and star today
and I am no longer sure what the real issue is. Perhaps
some of you can clarify. Here's the test I ran:

There are many a ways to get the same thing done, but I
did it in a most intuitive way (to me). I had 3 windows
to the same directory. In one window, I ran a simple
shell program interactively and continuously:
while true
do
touch big.inputfile
done

And in the 2nd window, I ran a command:
tar cf ../moo.tar . > tar_errlog 2>&1

There are a bunch of files in this directory and below,
but one of them (big.inputfile) is 4.7gb. The above
tar command exited with exit status 0 and a line like:
tar: ./input.big: file changed as we read it
The tar I used is version 1.14.

Also, in the 3rd window, I ran a command:
star cf ../moo.star . > star_errlog 2>&1

Everything else remained the same. Here, start exited
with 0 as well, while it gave me this line in errlog:
star: 418759 blocks + 0 bytes (total of 4288092160 bytes = 4187590.00k).

The star version is 1.5a25.

Here's what started this very thread:

> Some backup tools that you might wish to use emit warnings or errors
> if the files they are trying to copy change while the copy proceeds.
> This situation is normal, and not an error, when taking a base backup
> of an active database; so you need to ensure that you can distinguish
> complaints of this sort from real errors. For example, some versions
> of rsync return a separate exit code for "vanished source files", and
> you can write a driver script to accept this exit code as a non-error
> case. Also, some versions of GNU tar consider it an error if a file is
> changed while tar is copying it. There does not seem to be any very
> convenient way to distinguish this error from other types of errors,
> other than manual inspection of tar's messages. GNU tar is therefore
> not the best tool for making base backups.

Is the issue to do with the exit code of the backup program? If that's
the case, both tar and star (at least with these very versions) are not
acceptable. Or is the issue to do with what the program spits out as
human readable message, which should be or shouldn't be ignored?

Another big problem I see is that neither program lists the exit codes
in the man pages. That's not good. Rsync man page has a section where
all exit code is listed, but I have no foreign drive on this host. I
am tempted to get a tiny machine just to do the backup.

I'd be interested your insights and thoughts.

Regards,

Tena


-----Original Message-----
From: Kenneth Marshall [mailto:ktm@rice.edu]
Sent: Wed 8/22/2007 5:30 AM
To: Tena Sakai
Cc: Kevin Grittner; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] tar, but not gnu tar

Tena,

We have been very happy with star. It is a very nice pax,
cpio, gnutar,... replacement. You may want to give it a try.

Ken

On Tue, Aug 21, 2007 at 06:53:53PM -0700, Tena Sakai wrote:
> Hi Kevin,
>
> Yes, I have, but I am much more familiar with tar.
> I think I will go with the latest gnu tar (v 1.18)
> which is suggested by Bruce. I will play with it
> tomorrow and see how it goes.
>
> Thanks.
>
> Tena
>
> tsakai@gallo.ucsf.edu
>
>
> -----Original Message-----
> From: Kevin Grittner [mailto:Kevin.Grittner@wicourts.gov]
> Sent: Tue 8/21/2007 6:02 PM
> To: Tena Sakai; pgsql-admin@postgresql.org
> Subject: Re: [ADMIN] tar, but not gnu tar
>
> >>> On Tue, Aug 21, 2007 at 7:28 PM, in message
> <FE44E0D7EAD2ED4BB2165071DB8E328C03062B59@egcrc-ex01.egcrc.org>, "Tena Sakai"
> <tsakai@gallo.ucsf.edu> wrote:
> >
> > On my linux machine, gnu tar is the tar. Does
> > anybody have a suggestion as to where I can go
> > to get a tar that is not gnu?
>
> Have you considered using cpio instead?
>
> http://www.gnu.org/software/cpio/
>
> -Kevin
>
>
>
>

Tena Sakai wrote:
> Hi Everybody,
>
> I had a bit of time to experiment with tar and star today
> and I am no longer sure what the real issue is. Perhaps
> some of you can clarify. Here's the test I ran:
>
> There are many a ways to get the same thing done, but I
> did it in a most intuitive way (to me). I had 3 windows
> to the same directory. In one window, I ran a simple
> shell program interactively and continuously:
> while true
> do
> touch big.inputfile
> done

I don't think 'touch' is enough for tar to see the file as changed (you
are only updating metadata). (tar did complain but the file contents
didn't so it is hard to say if that is a good test.) You should change
the file contents during the backup.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

Bruce Momjian <bruce@momjian.us> writes:
> Tena Sakai wrote:
>> I had a bit of time to experiment with tar and star today
>> and I am no longer sure what the real issue is. Perhaps
>> some of you can clarify. Here's the test I ran:

> I don't think 'touch' is enough for tar to see the file as changed (you
> are only updating metadata). (tar did complain but the file contents
> didn't so it is hard to say if that is a good test.) You should change
> the file contents during the backup.

In fact, I'll bet that you have to change the file *length* during the
backup to trigger gnu tar's complaint. If it were rigorously checking
for file content change, it'd have to read the whole of every file
twice, which hardly seems like overhead that anyone would accept. But a
check for length change would just mean one extra stat() call per file,
which is a whole lot more plausible.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

Hi Tom,
Hi Bruce,

Thanks for your input. Fair enough. I redid it.
This time, in the first window, I ran this mini
shell program continuously:

while true
> do
> date >> big.input
> done

In the 2nd window:

tar cf ../moo.tar . > tar_errlog 2>&1

Tar returned the exit code 0. In the tar_errlog, it said:
tar: ./big.input: file changed as we read it

In the 3rd:

star cf ../moo.star . > star_errlog 2>&1

Star returned exit code of 254. 6 lines were in
star_errlog:
star: 'big.input': file changed size (increased).
star: 419190 blocks + 0 bytes (total of 4292505600 bytes = 4191900.00k).
star: The following problems occurred during archive processing:
star: Cannot: stat 0, open 0, read/write 0. Size changed 1.
star: Missing links 0, Name too long 0, File too big 0, Not dumped 0.
star: Processed all possible files, despite earlier errors.

That was so different from the previous test, I ran
star one more time, but this time I killed the process
in the first window. Ie., I ran the same star test while
nothing was changing. It ruturned exit code 0, and a line
in the star_errlog:
star: 419467 blocks + 0 bytes (total of 4295342080 bytes = 4194670.00k).

I am inclined to use star. At the bottom of the star man page,
the author Joerg Schilling gives out his email address (3 of
them!) and I will email him and ask if he can supply a list of
all exit codes.

Regards,

Tena Sakai
tsakai@gallo.ucsf.edu



-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Wed 8/22/2007 10:18 PM
To: Bruce Momjian
Cc: Tena Sakai; Kenneth Marshall; Kevin Grittner; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] tar, but not gnu tar

Bruce Momjian <bruce@momjian.us> writes:
> Tena Sakai wrote:
>> I had a bit of time to experiment with tar and star today
>> and I am no longer sure what the real issue is. Perhaps
>> some of you can clarify. Here's the test I ran:

> I don't think 'touch' is enough for tar to see the file as changed (you
> are only updating metadata). (tar did complain but the file contents
> didn't so it is hard to say if that is a good test.) You should change
> the file contents during the backup.

In fact, I'll bet that you have to change the file *length* during the
backup to trigger gnu tar's complaint. If it were rigorously checking
for file content change, it'd have to read the whole of every file
twice, which hardly seems like overhead that anyone would accept. But a
check for length change would just mean one extra stat() call per file,
which is a whole lot more plausible.

regards, tom lane