Tom Lane wrote:
> I wrote:
>
>>I'm off for a little visit with oprofile...
>
>
> It seems the answer is that fwrite() does have pretty significant
> per-call overhead, at least on Fedora Core 4. The patch I did yesterday
> still ended up making an fwrite() call every few characters when dealing
> with bytea text output, because it'd effectively do two fwrite()s per
> occurrence of '\' in the data being output. I've committed a further
> hack that buffers a whole data row before calling fwrite(). Even though
> this presumably is adding one extra level of data copying, it seems to
> make things noticeably faster:

(semi-OT) This recoding seems like a perfect preparation for a third
COPY format, compressed.

>
> Let me know what this does on your Debian machine ...

Takes a while, need a different kernel booted because the current isn't
oprofile ready.

Regards,
Andreas

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

"Marko Kreen" <markokr@gmail.com> writes:
> If you want to use fwrite as string operator, then maybe
> should replace it with fwrite_unlocked?

ISTM that in a single-threaded application such as the backend,
it should be libc's responsibility to avoid such overhead, not
ours.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

On 5/27/06, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> "Marko Kreen" <markokr@gmail.com> writes:
> > If you want to use fwrite as string operator, then maybe
> > should replace it with fwrite_unlocked?
>
> ISTM that in a single-threaded application such as the backend,
> it should be libc's responsibility to avoid such overhead, not
> ours.

Obviously, except glibc guys seems to be philosophically
opposed to this, so apps need to work around it.

AFAIK at least *BSDs have got this right, don't know
about others.

--
marko

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faq

On Sat, May 27, 2006 at 06:36:15PM +0300, Marko Kreen wrote:
> >ISTM that in a single-threaded application such as the backend,
> >it should be libc's responsibility to avoid such overhead, not
> >ours.
>
> Obviously, except glibc guys seems to be philosophically
> opposed to this, so apps need to work around it.
>
> AFAIK at least *BSDs have got this right, don't know
> about others.

Given there is no way to know if you're running single threaded or not,
I don't think glibc can take chances like that.

In any case, this isn't the issue here. Glibc doesn't do any locking
unless pthread is linked in. Ofcourse, it takes a few cycles to
determine that, but I don't think that'd cause a major slowdown.

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFEeH4EIB7bNG8LQkwRAhiMAJwPy3LNeJFNRep8ALJQvh JYH9ORrwCfUX1V
j2k2gFGQVKCfM7kDdOAr6rY=
=ZnNv
-----END PGP SIGNATURE-----

On 5/27/06, Martijn van Oosterhout <kleptog@svana.org> wrote:
> Given there is no way to know if you're running single threaded or not,
> I don't think glibc can take chances like that.

There's CPP symbol _REENTRANT for that and in run time,
libc can detect call to pthread_create [1].

> In any case, this isn't the issue here. Glibc doesn't do any locking
> unless pthread is linked in. Ofcourse, it takes a few cycles to
> determine that, but I don't think that'd cause a major slowdown.

You are conflicting with your previous paragraph

Otherwise you are right - that how a libc obviously should work, right?

http://marc.theaimsgroup.com/?l=glib...5741325472&w=2
http://marc.theaimsgroup.com/?l=glib...0641923178&w=2

I did a small test that does several fputc calls to /dev/null,
with various workarounds:

* lock.enabled is standard app.
* lock.disabled calls __fsetlocking(FSETLOCKING_BYCALLER),
as suggested by Ulrich Drepper.
* lock.unlocked calls fputc_unlocked

lock.enabled 48s
lock.disabled 28s
lock.unlocked 25s

I attached the test, you can measure yourself.

So I prepared a patch that calls __fsetlocking() in AllocateFile.
Andreas, Tom could you measure if it makes any difference?

--
marko

[1] In the first thread I linked, there was very clever
optimisation proposed using this function, that would
quarantee thread-safety even without _REENTRANT.

Unfortunately, event if U. Drepper changes his mind someday
and fixes the locking for singe-threaded apps, it would
very likely break binary compatibility with old apps,
so it wont happen in the near future.


---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings

On Sat, May 27, 2006 at 08:23:35PM +0300, Marko Kreen wrote:
> On 5/27/06, Martijn van Oosterhout <kleptog@svana.org> wrote:
> >Given there is no way to know if you're running single threaded or not,
> >I don't think glibc can take chances like that.
>
> There's CPP symbol _REENTRANT for that and in run time,
> libc can detect call to pthread_create [1].

There are a number of way to create threads, not all of which involve
pthread_create. I think my point is that you are not required to
declare _REENTRANT to get reentrant functions and there is no
_NOTREENTRANT symbol you can define.

> I did a small test that does several fputc calls to /dev/null,
> with various workarounds:

All your test proved was that it took 20 nanoseconds in each call to
fputc to determine no locking was required. I don't know how fast your
machine is, but thats probably just a few cycles. A better example
would be if there was actually some locking going on, i.e. add
-lpthread to the compile line. On my machine I get:

No -lpthread
lock.enabled 91s
lock.disabled 50s
lock.unlocked 36s

With -lpthread
lock.enabled 323s
lock.disabled 50s
lock.unlocked 36s

So yes, if you can guarentee no locking is required and tell glibc
that, you get optimal performace. But the *default* is to play it safe
and take a few extra cycles to check if locking is required at all.
Better than locking all the time wouldn't you agree? Just because your
app didn't declare _REENTRANT doesn't mean any of the libraries it
uses didn't.

The crux of the matter is though, if you're calling something a million
times, you're better off trying to find an alternative anyway. There is
a certain amount of overhead to calling shared libraries and no amount
of optimisation of the library is going save you that.

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFEeMvLIB7bNG8LQkwRAtZOAKCCUOnT9hiHQwX4EG36Xm m2L9zmfQCfQhuH
Z12HW+3B6S6VKrmBzKsPnzo=
=cVYD
-----END PGP SIGNATURE-----

On 5/28/06, Martijn van Oosterhout <kleptog@svana.org> wrote:
> With -lpthread
> lock.enabled 323s
> lock.disabled 50s
> lock.unlocked 36s

I forgot to test with -lpthread, my bad. Indeed by default
something less expensive that full locking is going on.

> The crux of the matter is though, if you're calling something a million
> times, you're better off trying to find an alternative anyway. There is
> a certain amount of overhead to calling shared libraries and no amount
> of optimisation of the library is going save you that.

The crux of the matter was if its possible to use fwrite
as easy string combining mechanism and the answer is no,
because it's not lightweight enough.

--
marko

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

Marko Kreen wrote:
> On 5/28/06, Martijn van Oosterhout <kleptog@svana.org> wrote:
>> With -lpthread
>> lock.enabled 323s
>> lock.disabled 50s
>> lock.unlocked 36s
>
> I forgot to test with -lpthread, my bad. Indeed by default
> something less expensive that full locking is going on.
>
>> The crux of the matter is though, if you're calling something a million
>> times, you're better off trying to find an alternative anyway. There is
>> a certain amount of overhead to calling shared libraries and no amount
>> of optimisation of the library is going save you that.
>
> The crux of the matter was if its possible to use fwrite
> as easy string combining mechanism and the answer is no,
> because it's not lightweight enough.
>
IIRC the windows port make use of multi-threading to simulate signals and it's likely that
some add-on modules will bring in libs like pthread. It would be less ideal if PostgreSQL
was designed to take a significant performance hit when that happens. Especially if a viable
alternative exists.

Regards,
Thomas Hallgren


---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

Marko Kreen wrote:
> On 5/28/06, Martijn van Oosterhout <kleptog@svana.org> wrote:
> > With -lpthread
> > lock.enabled 323s
> > lock.disabled 50s
> > lock.unlocked 36s
>
> I forgot to test with -lpthread, my bad. Indeed by default
> something less expensive that full locking is going on.
>
> > The crux of the matter is though, if you're calling something a million
> > times, you're better off trying to find an alternative anyway. There is
> > a certain amount of overhead to calling shared libraries and no amount
> > of optimisation of the library is going save you that.
>
> The crux of the matter was if its possible to use fwrite
> as easy string combining mechanism and the answer is no,
> because it's not lightweight enough.

So your patch to src/backend/storage/file/fd.c should be discarded? OK.

--
Bruce Momjian http://candle.pha.pa.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

On 5/30/06, Bruce Momjian <pgman@candle.pha.pa.us> wrote:
> > The crux of the matter was if its possible to use fwrite
> > as easy string combining mechanism and the answer is no,
> > because it's not lightweight enough.
>
> So your patch to src/backend/storage/file/fd.c should be discarded? OK.

Yes. It was just for experimenting. As I understand Tom
already rewrote the critical path.

--
marko

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly