Please consider this mail as submission of PL/Proxy into core.

Documentation:
https://developer.skype.com/SkypeGar...ojects/PlProxy

Previous submission with overview:
http://archives.postgresql.org/pgsql...3/msg01763.php

- The code has changed rather little compared to first submission,
biggest changes have been select()->poll() conversion and
making scanner work with flex 2.5.4.

- My interest in replacing dblink has decreased, it is for
-hackers to decide if this is interesting goal or not.

The patch itself:
http://plproxy.projects.postgresql.o...oxy-v1.diff.gz

I have not much effort into patch, except made it compile
in with main source. I'd like to get general consensus that
the idea is acceptable first.

Following are major parts that need work before committing:

- SGML documentation.
- Makefile review.
- Integrate regression tests into main test framework.

There are 2 things that could be migrated in to core code,
but this can be done later:

- Compat poll() function based on select() - can be used to get
rid of #ifdef HAVE_POLL in various places in core code.

- rowstamp.h - all the PL-s could use it to detect row changes.

There are few code beautification ideas on which I'd like to
get feedback from wider audience:

- Drop CONNECT statement. This was added to make creating
simple dblink style calls easier, but actually its not
maintainable compared to CLUSTER, it can be used only
for occasional hacks. OTOH, if we want to deprecate
dblink and replace it with PL/Proxy, it should probably stay.

- Drop SELECT statement. This was added as it was simple to do
and also to be straightforward replacement for dblink. But
it's conceptually ugly. Also it gives impression that there
will be UPDATE, DELETE and IF... which will not happen.

The good developement strategy when using plproxy is:

1. Create regular database with function-based API,
keeping an eye that interfaces would support
partitioning.
2. If load grows, split database to several pieces,
directing clients to db where all functions are
plproxy ones which will rediect to correct
partition.

Having the SELECT available discourages good design
where all partitions are functional on their own.

--
marko

Diffstat for the patch:
src/include/catalog/pg_pltemplate.h | 1
src/pl/Makefile | 2
src/pl/plproxy/Makefile | 97 +++
src/pl/plproxy/cluster.c | 542 ++++++++++++++++++
src/pl/plproxy/execute.c | 750 +++++++++++++++++++++++++
src/pl/plproxy/expected/plproxy_clustermap.out | 71 ++
src/pl/plproxy/expected/plproxy_errors.out | 66 ++
src/pl/plproxy/expected/plproxy_init.out | 1
src/pl/plproxy/expected/plproxy_many.out | 116 +++
src/pl/plproxy/expected/plproxy_select.out | 37 +
src/pl/plproxy/expected/plproxy_test.out | 266 ++++++++
src/pl/plproxy/function.c | 408 +++++++++++++
src/pl/plproxy/main.c | 218 +++++++
src/pl/plproxy/parser.y | 190 ++++++
src/pl/plproxy/plproxy.h | 314 ++++++++++
src/pl/plproxy/poll_compat.c | 140 ++++
src/pl/plproxy/poll_compat.h | 58 +
src/pl/plproxy/query.c | 299 +++++++++
src/pl/plproxy/result.c | 222 +++++++
src/pl/plproxy/rowstamp.h | 27
src/pl/plproxy/scanner.l | 329 ++++++++++
src/pl/plproxy/sql/plproxy_clustermap.sql | 56 +
src/pl/plproxy/sql/plproxy_errors.sql | 63 ++
src/pl/plproxy/sql/plproxy_init.sql | 63 ++
src/pl/plproxy/sql/plproxy_many.sql | 66 ++
src/pl/plproxy/sql/plproxy_select.sql | 37 +
src/pl/plproxy/sql/plproxy_test.sql | 164 +++++
src/pl/plproxy/type.c | 308 ++++++++++
28 files changed, 4909 insertions(+), 2 modifications(!)

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Wednesday 14 May 2008 13:29, Marko Kreen wrote:
> - SGML documentation.
> - Makefile review.
> - Integrate regression tests into main test framework.

Has PL/proxy been tested on other OSes? FreeBSD/Solaris/Windows?

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 5/15/08, Josh Berkus <josh@agliodbs.com> wrote:
> On Wednesday 14 May 2008 13:29, Marko Kreen wrote:
> > - SGML documentation.
> > - Makefile review.
> > - Integrate regression tests into main test framework.
>
> Has PL/proxy been tested on other OSes? FreeBSD/Solaris/Windows?

It definitely works on Linux and MacOS. I've seen ports for *BSD.
I think any unix-like OS-es should work fine.

In fact, only syscalls it does on its own are gettimeofday() and poll()
[or select()], otherwise it calls either core or libpq functions.
So I see no reason why it shouldnt already work on Windows.

The biggest portability problem thus far has been scanner.l, which at
the beginning was written for flex 2.5.33+, but 2.5.4 is still pretty
widespread. But this I fixed in 2.0.3.

Hmm.. Now that I think about it, in my effort to remove malloc() calls
in both scanner and parser I told bison to use alloca(). Is it portability
concern? Easy to fix, just need to be careful not to create memleaks.

--
marko

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

"Marko Kreen" <markokr@gmail.com> writes:
> Hmm.. Now that I think about it, in my effort to remove malloc() calls
> in both scanner and parser I told bison to use alloca(). Is it portability
> concern?

Yes.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 5/15/08, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> "Marko Kreen" <markokr@gmail.com> writes:
> > Hmm.. Now that I think about it, in my effort to remove malloc() calls
> > in both scanner and parser I told bison to use alloca(). Is it portability
> > concern?
>
> Yes.

How about following patch? I have bison 2.3 and it seems not to do
global allocation, so it should be fine. There may be early exit
with elog(ERRROR) inside so I'd like to avoid malloc() itself.

Is there some older bison that keeps allocations around?
They would need bit more work...

--- src/parser.y 14 May 2008 12:25:00 -0000 1.7
+++ src/parser.y 15 May 2008 07:34:53 -0000
@@ -24,7 +24,9 @@
void plproxy_yy_scan_bytes(const char *bytes, int len);

/* avoid permanent allocations */
-#define YYSTACK_USE_ALLOCA 1
+#define YYMALLOC palloc
+#define YYFREE pfree
+
/* remove unused code */
#define YY_LOCATION_PRINT(File, Loc) (0)
#define YY_(x) (x)

I will roll new full patch when more comments have appeared.

--
marko

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 5/15/08, Marko Kreen <markokr@gmail.com> wrote:
> On 5/15/08, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > "Marko Kreen" <markokr@gmail.com> writes:
> > > Hmm.. Now that I think about it, in my effort to remove malloc() calls
> > > in both scanner and parser I told bison to use alloca(). Is it portability
> > > concern?
> >
> > Yes.
>
>
> How about following patch? I have bison 2.3 and it seems not to do
> global allocation, so it should be fine. There may be early exit
> with elog(ERRROR) inside so I'd like to avoid malloc() itself.
>
> Is there some older bison that keeps allocations around?
> They would need bit more work...
>
> --- src/parser.y 14 May 2008 12:25:00 -0000 1.7
> +++ src/parser.y 15 May 2008 07:34:53 -0000
> @@ -24,7 +24,9 @@
> void plproxy_yy_scan_bytes(const char *bytes, int len);
>
> /* avoid permanent allocations */
> -#define YYSTACK_USE_ALLOCA 1
> +#define YYMALLOC palloc
> +#define YYFREE pfree
> +
> /* remove unused code */
> #define YY_LOCATION_PRINT(File, Loc) (0)
> #define YY_(x) (x)
>
> I will roll new full patch when more comments have appeared.

Checked bison 1.875, and it does not use YYMALLOC/YYFREE.
But luckily its allocation pattern seems sane, so following should work:

--- src/parser.y 14 May 2008 12:25:00 -0000 1.7
+++ src/parser.y 15 May 2008 08:18:14 -0000
@@ -24,7 +24,9 @@
void plproxy_yy_scan_bytes(const char *bytes, int len);

/* avoid permanent allocations */
-#define YYSTACK_USE_ALLOCA 1
+#define malloc palloc
+#define free pfree
+
/* remove unused code */
#define YY_LOCATION_PRINT(File, Loc) (0)
#define YY_(x) (x)

--
marko

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

"Marko Kreen" <markokr@gmail.com> writes:
> How about following patch? I have bison 2.3 and it seems not to do
> global allocation, so it should be fine. There may be early exit
> with elog(ERRROR) inside so I'd like to avoid malloc() itself.

None of our other parsers fool with bison's memory allocation;
why does yours need to?

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 5/15/08, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> "Marko Kreen" <markokr@gmail.com> writes:
> > How about following patch? I have bison 2.3 and it seems not to do
> > global allocation, so it should be fine. There may be early exit
> > with elog(ERRROR) inside so I'd like to avoid malloc() itself.
>
> None of our other parsers fool with bison's memory allocation;
> why does yours need to?

Because that way I can be sure I understand their allocation behaviour.

Eg. how does src/backend/parser/gram.c not leak memory on syntax error?
I don't understand it.

But if I force them use palloc(), always, I can be sure memore is freed.

--
marko

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

"Marko Kreen" <markokr@gmail.com> writes:
> Eg. how does src/backend/parser/gram.c not leak memory on syntax error?

It's not a leak because the memory can be re-used during the next
command.

I believe you'll find that trying to make it use palloc is a failure
because it keeps static pointers that it expects will stay valid across
calls.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 5/15/08, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> "Marko Kreen" <markokr@gmail.com> writes:
> > Eg. how does src/backend/parser/gram.c not leak memory on syntax error?
>
> It's not a leak because the memory can be re-used during the next
> command.

I may be blind, but I don't see any static variables there.

> I believe you'll find that trying to make it use palloc is a failure
> because it keeps static pointers that it expects will stay valid across
> calls.

Thats true, I need to drop the redefines if the allocations may be reused.

--
marko

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers