Unix Technical Forum

Patch to log usage of temporary files

This is a discussion on Patch to log usage of temporary files within the Pgsql Patches forums, part of the PostgreSQL category; --> Thanks to Simon Riggs and Bruce for input that helped me put this together. -- Bill Moran Collaborative Fusion ...


Go Back   Unix Technical Forum > Database Server Software > PostgreSQL > Pgsql Patches

FAQ Members List Calendar Search Today's Posts Mark Forums Read
  #1 (permalink)  
Old 04-18-2008, 10:13 AM
Bill Moran
 
Posts: n/a
Default Patch to log usage of temporary files


Thanks to Simon Riggs and Bruce for input that helped me put this together.

--
Bill Moran
Collaborative Fusion Inc.



---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #2 (permalink)  
Old 04-18-2008, 10:13 AM
Alvaro Herrera
 
Posts: n/a
Default Re: Patch to log usage of temporary files

Bill Moran wrote:
>
> Thanks to Simon Riggs and Bruce for input that helped me put this together.


Please change things to save the stat() syscall when the feature is not
in use.

Nitpick: also note our brace placement convention (though this would be
fixed by pgindent, but still).

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #3 (permalink)  
Old 04-18-2008, 10:13 AM
Andrew Dunstan
 
Posts: n/a
Default Re: Patch to log usage of temporary files

Bill Moran wrote:
> + if (stat(vfdP->fileName, &filestats) == 0) {
> + if (trace_temp_files)
>

Shouldn't these tests be the other way around?

cheers

andrew

---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #4 (permalink)  
Old 04-18-2008, 10:13 AM
Bill Moran
 
Posts: n/a
Default Re: Patch to log usage of temporary files

In response to Alvaro Herrera <alvherre@commandprompt.com>:

> Bill Moran wrote:
> >
> > Thanks to Simon Riggs and Bruce for input that helped me put this together.

>
> Please change things to save the stat() syscall when the feature is not
> in use.


Do you have a suggestion on how to do that and still have the PG_TRACE1()
work? That was specifically requested by Simon Riggs.

I'm not at all familiar with how the PG_TRACE probes work, so I'd be
interested to hear suggestions on how to wrap that in an if. If I remove
the PG_TRACE, it becomes a simple matter to skip the stat() if the feature
is disabled.

> Nitpick: also note our brace placement convention (though this would be
> fixed by pgindent, but still).


Sorry, I thought I _was_ following the convention. Must have missed
something. Is there a written style guide somewhere? Might drive things
home a little better than just looking at other folks code.

--
Bill Moran
Collaborative Fusion Inc.

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #5 (permalink)  
Old 04-18-2008, 10:13 AM
Tom Lane
 
Posts: n/a
Default Re: Patch to log usage of temporary files

Bill Moran <wmoran@collaborativefusion.com> writes:
> In response to Alvaro Herrera <alvherre@commandprompt.com>:
>> Please change things to save the stat() syscall when the feature is not
>> in use.


> Do you have a suggestion on how to do that and still have the PG_TRACE1()
> work? That was specifically requested by Simon Riggs.


Well, we are NOT paying a stat() call on every single file close,
whether Simon wants it or not. PG_TRACE1 doesn't even do anything
on non-Solaris platforms, for pete's sake.

Perhaps it would be reasonable to define trace_temp_files as the minimum
file size to log; then you could do something like

if (trace_temp_files > 0)
{
if (stat(vfdP->fileName, &filestats) < 0)
elog(LOG, ...);
else
{
if (filestats.st_size / BLCKSZ >= trace_temp_files)
ereport(LOG, ...);
PG_TRACE1(temp__file__cleanup, filestats.st_size);
}
}

Note that elog(ERROR) is quite inappropriate here.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #6 (permalink)  
Old 04-18-2008, 10:13 AM
Simon Riggs
 
Posts: n/a
Default Re: [HACKERS] Patch to log usage of temporary files

On Tue, 2007-01-02 at 18:20 -0500, Tom Lane wrote:
> Bill Moran <wmoran@collaborativefusion.com> writes:
> > In response to Alvaro Herrera <alvherre@commandprompt.com>:
> >> Please change things to save the stat() syscall when the feature is not
> >> in use.

>
> > Do you have a suggestion on how to do that and still have the PG_TRACE1()
> > work? That was specifically requested by Simon Riggs.

>
> Well, we are NOT paying a stat() call on every single file close,
> whether Simon wants it or not.


Simon doesn't/wouldn't want the stat() call on each file close.

If you put the PG_TRACE macro outside of the if test, yet prior to the
file close, you can pass the filename through like this

PG_TRACE1(temp__file__cleanup, vfdP->fileName);

That way DTrace can make its own call to find out filesize, if it would
like to... and we don't need to stat() before each temp file close.
That's much more flexible and useful, as well as better performance.

--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com



---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #7 (permalink)  
Old 04-18-2008, 10:13 AM
Bill Moran
 
Posts: n/a
Default Re: [HACKERS] Patch to log usage of temporary files

In response to "Simon Riggs" <simon@2ndquadrant.com>:

> On Tue, 2007-01-02 at 18:20 -0500, Tom Lane wrote:
> > Bill Moran <wmoran@collaborativefusion.com> writes:
> > > In response to Alvaro Herrera <alvherre@commandprompt.com>:
> > >> Please change things to save the stat() syscall when the feature is not
> > >> in use.

> >
> > > Do you have a suggestion on how to do that and still have the PG_TRACE1()
> > > work? That was specifically requested by Simon Riggs.

> >
> > Well, we are NOT paying a stat() call on every single file close,
> > whether Simon wants it or not.

>
> Simon doesn't/wouldn't want the stat() call on each file close.
>
> If you put the PG_TRACE macro outside of the if test, yet prior to the
> file close, you can pass the filename through like this
>
> PG_TRACE1(temp__file__cleanup, vfdP->fileName);
>
> That way DTrace can make its own call to find out filesize, if it would
> like to... and we don't need to stat() before each temp file close.
> That's much more flexible and useful, as well as better performance.


OK, I think I've managed to adjust this patch so that everyone is happy
and it's better as well.

* PG_TRACE will work whether the GUC var is enabled or not, it sends the
fileName, as suggested by Simon
* stat() call is not made if trace_temp_files is disabled
* trace_temp_files is now an int: -1 disables, 0 and up equate to "log if
the file is this size or larger"
* Cleaned things up a bit -- should be more in line with PostgreSQL
coding standards
* failed stat is reported as LOG instead of ERROR

Done a bit of testing here, and everything seems to be in order.

--
Bill Moran
Collaborative Fusion Inc.


---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #8 (permalink)  
Old 04-18-2008, 10:13 AM
Andrew Dunstan
 
Posts: n/a
Default Re: [HACKERS] Patch to log usage of temporary files

Bill Moran wrote:
> + if (trace_temp_files != -1)
>


Might be more robust to say

if (trace_temp_files >= 0)

cheers

andrew

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #9 (permalink)  
Old 04-18-2008, 10:13 AM
Bill Moran
 
Posts: n/a
Default Re: [HACKERS] Patch to log usage of temporary files

Andrew Dunstan <andrew@dunslane.net> wrote:
>
> Bill Moran wrote:
> > + if (trace_temp_files != -1)
> >

>
> Might be more robust to say
>
> if (trace_temp_files >= 0)


Because it would allow for the easy addition of more negative numbers
with magic value?

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #10 (permalink)  
Old 04-18-2008, 10:13 AM
Andrew Dunstan
 
Posts: n/a
Default Re: [HACKERS] Patch to log usage of temporary files

Bill Moran wrote:
> Andrew Dunstan <andrew@dunslane.net> wrote:
>>
>> Bill Moran wrote:
>> > + if (trace_temp_files != -1)
>> >

>>
>> Might be more robust to say
>>
>> if (trace_temp_files >= 0)

>
> Because it would allow for the easy addition of more negative numbers
> with magic value?
>



because ISTM any negative number here should mean no action is to be
taken. Otherwise how else is it different from 0?

cheers

andrew


---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On
Forum Jump


All times are GMT. The time now is 05:01 PM.


Powered by vBulletin® Version 3.6.5
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.2.0
www.UnixAdminTalk.com