Jim C. Nasby wrote:
> Moving to -hackers.
>
> On Fri, May 11, 2007 at 04:37:44PM +0100, Heikki Linnakangas wrote:
>>> If you know when the checkpoint ended, and you know how long each of the
>>> pieces took, you can reconstruct the other times easily. The way you
>>> describe this it is true--that the summary is redundant given the
>>> detail--but if you put yourself in the shoes of a log file parser the
>>> other way around is easier to work with. Piecing together log entries
>>> is a pain, splitting them is easy.
>>>
>>> If I had to only keep one line out of this, it would be the one with the
>>> summary. It would be nice to have it logged at INFO.
>> Yeah, if we have the summary line we don't need the other lines and vice
>> versa. I have sympathy for parsing log files, I've done that a lot in
>> the past and I can see what you mean. Having the individual lines is
>> nice when you're monitoring a running system; you don't get the summary
>> line until the checkpoint is finished. I suppose we can have both the
>> individual lines and the summary, the extra lines shouldn't hurt anyone,
>> and you won't get them unless you turn on the new log_checkpoints
>> parameter anyway.
>
> Not to beat a dead horse, but do we really want to force folks to be
> parsing logs for performance monitoring? Especially if that log parsing
> is just going to result in data being inserted into a table anyway?
>
> I know there's concern about performance of the stats system and maybe
> that needs to be addressed, but pushing users to log parsing is a lot of
> extra effort, non-standard, likely to be overlooked, and doesn't play
> well with other tools. It also conflicts with all the existing
> statistics framework.

There is two counters for checkpoints in pgstats, the number of timed
(triggered by checkpoint_timeout) and requested (triggered by
checkpoint_segments) checkpoints.

Maybe we should improve the stats system so that we can collect events
with timestamps and durations, but in my experience log files actually
are the most reliable and universal way to collect real-time performance
information. Any serious tool has a generic log parser. The other
alternative is SNMP. I welcome the efforts on pgsnmpd..

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

Heikki Linnakangas wrote:
>>> Yeah, if we have the summary line we don't need the other lines and
>>> vice versa. I have sympathy for parsing log files, I've done that a
>>> lot in the past and I can see what you mean. Having the individual
>>> lines is nice when you're monitoring a running system; you don't get
>>> the summary line until the checkpoint is finished. I suppose we can
>>> have both the individual lines and the summary, the extra lines
>>> shouldn't hurt anyone, and you won't get them unless you turn on the
>>> new log_checkpoints parameter anyway.
>>
>> Not to beat a dead horse, but do we really want to force folks to be
>> parsing logs for performance monitoring? Especially if that log parsing
>> is just going to result in data being inserted into a table anyway?
>>
>> I know there's concern about performance of the stats system and maybe
>> that needs to be addressed, but pushing users to log parsing is a lot of
>> extra effort, non-standard, likely to be overlooked, and doesn't play
>> well with other tools. It also conflicts with all the existing
>> statistics framework.
>
> There is two counters for checkpoints in pgstats, the number of timed
> (triggered by checkpoint_timeout) and requested (triggered by
> checkpoint_segments) checkpoints.
>
> Maybe we should improve the stats system so that we can collect events
> with timestamps and durations, but in my experience log files actually
> are the most reliable and universal way to collect real-time performance
> information. Any serious tool has a generic log parser. The other
> alternative is SNMP. I welcome the efforts on pgsnmpd..

pgsnmpd can't provide any information that's not in the backend. Unless
we'd turn it into a log parser, which is not really something I think is
a good idea.

Log files are great for one kind of thing, "live statistics" through
SNMP or the statistics collector for another kind. It only gets wrong
when you put them in the wrong place. Things you poll regularly makes a
lot more sense in some kind of live view than in a log file.

//Magnus

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

On Sun, May 13, 2007 at 07:54:20AM +0100, Heikki Linnakangas wrote:
> Maybe we should improve the stats system so that we can collect events
> with timestamps and durations, but in my experience log files actually
> are the most reliable and universal way to collect real-time performance
> information. Any serious tool has a generic log parser. The other
> alternative is SNMP. I welcome the efforts on pgsnmpd..

Having timing information in the stats system would be useful, but I'm
not sure how it could actually be done. But at least if the information
is in the stats system it's easy to programatically collect and process.
SNMP is just one example of that (fwiw I agree with Magnus that it
probably doesn't make sense to turn pgsnmpd into a log parser...)
--
Jim Nasby decibel@decibel.org
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

Heikki Linnakangas napsal(a):
> Jim C. Nasby wrote:

>
> There is two counters for checkpoints in pgstats, the number of timed
> (triggered by checkpoint_timeout) and requested (triggered by
> checkpoint_segments) checkpoints.
>
> Maybe we should improve the stats system so that we can collect events
> with timestamps and durations, but in my experience log files actually
> are the most reliable and universal way to collect real-time performance
> information. Any serious tool has a generic log parser. The other
> alternative is SNMP. I welcome the efforts on pgsnmpd..

Whats about add some DTrace probes?

Zdenek


---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster