Unix Technical Forum

SEO

vBulletin Search Engine Optimization


Go Back   Unix Technical Forum > Database Server Software > PostgreSQL > Pgsql Performance

Register FAQ Members List Calendar Search Today's Posts Mark Forums Read
  #1 (permalink)  
Old 05-07-2008, 10:18 AM
Justin
 
Posts: n/a
Default need to speed up query

i've had to write queries to get trail balance values out of the GL
transaction table and i'm not happy with its performance

The table has 76K rows growing about 1000 rows per working day so the
performance is not that great it takes about 20 to 30 seconds to get all
the records for the table and when we limit it to single accounting
period it drops down to 2 seconds

Here is the query and explain . PostgreSql is 8.3.1 on new server with
raid 10 Serial SCSI.

SELECT period.period_id,
period.period_start,
period.period_end,
accnt.accnt_id,
accnt.accnt_number,
accnt.accnt_descrip,
period.period_yearperiod_id,
accnt.accnt_type,
COALESCE(( SELECT sum(gltrans.gltrans_amount) AS sum
FROM gltrans
WHERE gltrans.gltrans_date < period.period_start
AND gltrans.gltrans_accnt_id = accnt.accnt_id
AND gltrans.gltrans_posted = true), 0.00)::text::money AS
beginbalance,
COALESCE(( SELECT sum(gltrans.gltrans_amount) AS sum
FROM gltrans
WHERE gltrans.gltrans_date <= period.period_end
AND gltrans.gltrans_date >= period.period_start
AND gltrans.gltrans_amount <= 0::numeric
AND gltrans.gltrans_accnt_id = accnt.accnt_id
AND gltrans.gltrans_posted = true), 0.00)::text::money AS
negative,
COALESCE(( SELECT sum(gltrans.gltrans_amount) AS sum
FROM gltrans
WHERE gltrans.gltrans_date <= period.period_end
AND gltrans.gltrans_date >= period.period_start
AND gltrans.gltrans_amount >= 0::numeric
AND gltrans.gltrans_accnt_id = accnt.accnt_id
AND gltrans.gltrans_posted = true), 0.00)::text::money AS
positive,
COALESCE(( SELECT sum(gltrans.gltrans_amount) AS sum
FROM gltrans
WHERE gltrans.gltrans_date <= period.period_end
AND gltrans.gltrans_date >= period.period_start
AND gltrans.gltrans_accnt_id = accnt.accnt_id
AND gltrans.gltrans_posted = true), 0.00)::text::money AS
difference,
COALESCE(( SELECT sum(gltrans.gltrans_amount) AS sum
FROM gltrans
WHERE gltrans.gltrans_date <= period.period_end
AND gltrans.gltrans_accnt_id = accnt.accnt_id
AND gltrans.gltrans_posted = true), 0.00)::text::money AS
endbalance
FROM period, accnt
ORDER BY period.period_id, accnt.accnt_number;

"Sort (cost=4083970.56..4083974.89 rows=1729 width=57) (actual
time=24680.402..24681.386 rows=1729 loops=1)"
" Sort Key: period.period_id, accnt.accnt_number"
" Sort Method: quicksort Memory: 292kB"
" -> Nested Loop (cost=1.14..4083877.58 rows=1729 width=57) (actual
time=4.043..24674.258 rows=1729 loops=1)"
" -> Seq Scan on accnt (cost=0.00..4.33 rows=133 width=41)
(actual time=0.011..0.158 rows=133 loops=1)"
" -> Materialize (cost=1.14..1.27 rows=13 width=16) (actual
time=0.001..0.010 rows=13 loops=133)"
" -> Seq Scan on period (cost=0.00..1.13 rows=13
width=16) (actual time=0.005..0.023 rows=13 loops=1)"
" SubPlan"
" -> Aggregate (cost=1093.64..1093.65 rows=1 width=8) (actual
time=6.039..6.039 rows=1 loops=1729)"
" -> Bitmap Heap Scan on gltrans (cost=398.21..1092.18
rows=585 width=8) (actual time=5.171..5.623 rows=428 loops=1729)"
" Recheck Cond: ((gltrans_accnt_id = $1) AND
(gltrans_date <= $3))"
" Filter: gltrans_posted"
" -> BitmapAnd (cost=398.21..398.21 rows=636
width=0) (actual time=5.158..5.158 rows=0 loops=1729)"
" -> Bitmap Index Scan on
gltrans_gltrans_accnt_id_idx (cost=0.00..30.57 rows=1908 width=0)
(actual time=0.078..0.078 rows=574 loops=1729)"
" Index Cond: (gltrans_accnt_id = $1)"
" -> Bitmap Index Scan on
gltrans_gltrans_date_idx (cost=0.00..367.10 rows=25446 width=0) (actual
time=7.407..7.407 rows=63686 loops=1183)"
" Index Cond: (gltrans_date <= $3)"
" -> Aggregate (cost=58.19..58.20 rows=1 width=8) (actual
time=0.920..0.921 rows=1 loops=1729)"
" -> Bitmap Heap Scan on gltrans (cost=38.90..58.16
rows=9 width=8) (actual time=0.843..0.878 rows=40 loops=1729)"
" Recheck Cond: ((gltrans_date <= $3) AND
(gltrans_date >= $0) AND (gltrans_accnt_id = $1))"
" Filter: gltrans_posted"
" -> BitmapAnd (cost=38.90..38.90 rows=10
width=0) (actual time=0.839..0.839 rows=0 loops=1729)"
" -> Bitmap Index Scan on
gltrans_gltrans_date_idx (cost=0.00..8.08 rows=382 width=0) (actual
time=0.782..0.782 rows=5872 loops=1729)"
" Index Cond: ((gltrans_date <= $3) AND
(gltrans_date >= $0))"
" -> Bitmap Index Scan on
gltrans_gltrans_accnt_id_idx (cost=0.00..30.57 rows=1908 width=0)
(actual time=0.076..0.076 rows=574 loops=798)"
" Index Cond: (gltrans_accnt_id = $1)"
" -> Aggregate (cost=58.20..58.21 rows=1 width=8) (actual
time=0.897..0.898 rows=1 loops=1729)"
" -> Bitmap Heap Scan on gltrans (cost=38.89..58.19
rows=4 width=8) (actual time=0.845..0.874 rows=20 loops=1729)"
" Recheck Cond: ((gltrans_date <= $3) AND
(gltrans_date >= $0) AND (gltrans_accnt_id = $1))"
" Filter: (gltrans_posted AND (gltrans_amount >=
0::numeric))"
" -> BitmapAnd (cost=38.89..38.89 rows=10
width=0) (actual time=0.840..0.840 rows=0 loops=1729)"
" -> Bitmap Index Scan on
gltrans_gltrans_date_idx (cost=0.00..8.08 rows=382 width=0) (actual
time=0.783..0.783 rows=5872 loops=1729)"
" Index Cond: ((gltrans_date <= $3) AND
(gltrans_date >= $0))"
" -> Bitmap Index Scan on
gltrans_gltrans_accnt_id_idx (cost=0.00..30.57 rows=1908 width=0)
(actual time=0.077..0.077 rows=574 loops=798)"
" Index Cond: (gltrans_accnt_id = $1)"
" -> Aggregate (cost=58.20..58.21 rows=1 width=8) (actual
time=0.908..0.909 rows=1 loops=1729)"
" -> Bitmap Heap Scan on gltrans (cost=38.89..58.19
rows=4 width=8) (actual time=0.854..0.885 rows=20 loops=1729)"
" Recheck Cond: ((gltrans_date <= $3) AND
(gltrans_date >= $0) AND (gltrans_accnt_id = $1))"
" Filter: (gltrans_posted AND (gltrans_amount <=
0::numeric))"
" -> BitmapAnd (cost=38.89..38.89 rows=10
width=0) (actual time=0.843..0.843 rows=0 loops=1729)"
" -> Bitmap Index Scan on
gltrans_gltrans_date_idx (cost=0.00..8.08 rows=382 width=0) (actual
time=0.785..0.785 rows=5872 loops=1729)"
" Index Cond: ((gltrans_date <= $3) AND
(gltrans_date >= $0))"
" -> Bitmap Index Scan on
gltrans_gltrans_accnt_id_idx (cost=0.00..30.57 rows=1908 width=0)
(actual time=0.078..0.078 rows=574 loops=798)"
" Index Cond: (gltrans_accnt_id = $1)"
" -> Aggregate (cost=1093.64..1093.65 rows=1 width=8) (actual
time=5.485..5.485 rows=1 loops=1729)"
" -> Bitmap Heap Scan on gltrans (cost=398.21..1092.18
rows=585 width=8) (actual time=4.699..5.110 rows=388 loops=1729)"
" Recheck Cond: ((gltrans_accnt_id = $1) AND
(gltrans_date < $0))"
" Filter: gltrans_posted"
" -> BitmapAnd (cost=398.21..398.21 rows=636
width=0) (actual time=4.687..4.687 rows=0 loops=1729)"
" -> Bitmap Index Scan on
gltrans_gltrans_accnt_id_idx (cost=0.00..30.57 rows=1908 width=0)
(actual time=0.079..0.079 rows=574 loops=1729)"
" Index Cond: (gltrans_accnt_id = $1)"
" -> Bitmap Index Scan on
gltrans_gltrans_date_idx (cost=0.00..367.10 rows=25446 width=0) (actual
time=6.717..6.717 rows=57814 loops=1183)"
" Index Cond: (gltrans_date < $0)"
"Total runtime: 24682.580 ms"



--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #2 (permalink)  
Old 05-07-2008, 10:18 AM
Scott Marlowe
 
Posts: n/a
Default Re: need to speed up query

You're joining these two tables: period, accnt, but I'm not seeing an
on () clause or a where clause joining them. Is the cross product
intentional?

But what I'm seeing that seems like the lowest hanging fruit would be
two column indexes on the bits that are showing up in those bit map
scans. Like this part:

" Recheck Cond: ((gltrans_date <= $3) AND
(gltrans_date >= $0) AND gltrans_accnt_id = $1))"
" Filter: gltrans_posted"
" -> BitmapAnd (cost=38.90..38.90 rows=10
width=0) (actual time=0.839..0.839 rows=0 loops=1729)"
" -> Bitmap Index Scan on
gltrans_gltrans_date_idx (cost=0.00..8.08 rows=382 width=0) (actual
time=0.782..0.782 rows=5872 loops=1729)"
" Index Cond: ((gltrans_date <= $3)
AND (gltrans_date >= $0))"
" -> Bitmap Index Scan on
gltrans_gltrans_accnt_id_idx (cost=0.00..30.57 rows=1908 width=0)
(actual time=0.076..0.076 rows=574 loops=798)"
" Index Cond: (gltrans_accnt_id = $1)"

You are looking through 574 rows in one column and 5872 in another.
But when they're anded together, you get 0 rows. A two column index
there should really help.

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #3 (permalink)  
Old 05-07-2008, 10:18 AM
Gregory Williamson
 
Posts: n/a
Default Re: need to speed up query

Justin --

You wrote:
>
> i've had to write queries to get trail balance values out of the GL
> transaction table and i'm not happy with its performance
>
>
> The table has 76K rows growing about 1000 rows per working day so the
> performance is not that great it takes about 20 to 30 seconds to get all
> the records for the table and when we limit it to single accounting
> period it drops down to 2 seconds


So 30 seconds for 76 days (roughly) worth of numbers ? Not terrible but not great.

> Here is the query and explain . PostgreSql is 8.3.1 on new server with
> raid 10 Serial SCSI.

<... snipped 'cause I have a lame reader ...>

> " Sort Method: quicksort Memory: 292kB"

<...snip...>
> "Total runtime: 24682.580 ms"



I don't have any immediate thoughts but maybe you could post the table schemas and indexes. It looks to my untutored eye as if most of the estimates are fair so I am guessing that you have run analyze recently.

What is your sort memory set to ? If work_mem is too low then you'll go to disk (if you see tmp files under the postgres $PGDATA/base directory you might be seeing the result of this) ...

HTH

Greg Williamson
Senior DBA
DigitalGlobe

Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information and must be protected in accordance with those provisions. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

(My corporate masters made me say this.)


Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #4 (permalink)  
Old 05-07-2008, 10:18 AM
Justin
 
Posts: n/a
Default Re: need to speed up query

yes the cross join is intentional.

Thanks creating the two column index drop processing time to 15 to 17
seconds
put per period down to 1 second



Scott Marlowe wrote:
> You're joining these two tables: period, accnt, but I'm not seeing an
> on () clause or a where clause joining them. Is the cross product
> intentional?
>
> But what I'm seeing that seems like the lowest hanging fruit would be
> two column indexes on the bits that are showing up in those bit map
> scans. Like this part:
>
> " Recheck Cond: ((gltrans_date <= $3) AND
> (gltrans_date >= $0) AND gltrans_accnt_id = $1))"
> " Filter: gltrans_posted"
> " -> BitmapAnd (cost=38.90..38.90 rows=10
> width=0) (actual time=0.839..0.839 rows=0 loops=1729)"
> " -> Bitmap Index Scan on
> gltrans_gltrans_date_idx (cost=0.00..8.08 rows=382 width=0) (actual
> time=0.782..0.782 rows=5872 loops=1729)"
> " Index Cond: ((gltrans_date <= $3)
> AND (gltrans_date >= $0))"
> " -> Bitmap Index Scan on
> gltrans_gltrans_accnt_id_idx (cost=0.00..30.57 rows=1908 width=0)
> (actual time=0.076..0.076 rows=574 loops=798)"
> " Index Cond: (gltrans_accnt_id = $1)"
>
> You are looking through 574 rows in one column and 5872 in another.
> But when they're anded together, you get 0 rows. A two column index
> there should really help.
>
>


--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #5 (permalink)  
Old 05-07-2008, 10:18 AM
Justin
 
Posts: n/a
Default Re: need to speed up query

Gregory Williamson wrote:
>
> Justin --
>
> You wrote:
> >
> > i've had to write queries to get trail balance values out of the GL
> > transaction table and i'm not happy with its performance
> >
> >
> > The table has 76K rows growing about 1000 rows per working day so the
> > performance is not that great it takes about 20 to 30 seconds to get all
> > the records for the table and when we limit it to single accounting
> > period it drops down to 2 seconds

>
> So 30 seconds for 76 days (roughly) worth of numbers ? Not terrible
> but not great.
>
> > Here is the query and explain . PostgreSql is 8.3.1 on new server with
> > raid 10 Serial SCSI.

> <... snipped 'cause I have a lame reader ...>
>

not according to the bench marks i have done, which were posted a
couple of months ago.
>
>
> > " Sort Method: quicksort Memory: 292kB"

> <...snip...>
> > "Total runtime: 24682.580 ms"

>
>
> I don't have any immediate thoughts but maybe you could post the table
> schemas and indexes. It looks to my untutored eye as if most of the
> estimates are fair so I am guessing that you have run analyze recently.
>
> What is your sort memory set to ? If work_mem is too low then you'll
> go to disk (if you see tmp files under the postgres $PGDATA/base
> directory you might be seeing the result of this) ...
>

i need to look into work mem its set at 25 megs which is fine for most
work unless we get into the accounting queries which have to be more
complicated than they need to be because how some of the tables are laid
out which i did not lay out.
>
>
> HTH
>
> Greg Williamson
> Senior DBA
> DigitalGlobe
>
> Confidentiality Notice: This e-mail message, including any
> attachments, is for the sole use of the intended recipient(s) and may
> contain confidential and privileged information and must be protected
> in accordance with those provisions. Any unauthorized review, use,
> disclosure or distribution is prohibited. If you are not the intended
> recipient, please contact the sender by reply e-mail and destroy all
> copies of the original message.
>
> (My corporate masters made me say this.)
>


Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #6 (permalink)  
Old 05-07-2008, 10:18 AM
PFC
 
Posts: n/a
Default Re: need to speed up query


> i've had to write queries to get trail balance values out of the GL
> transaction table and i'm not happy with its performance The table has
> 76K rows growing about 1000 rows per working day so the performance is
> not that great it takes about 20 to 30 seconds to get all the records
> for the table and when we limit it to single accounting period it drops
> down to 2 seconds


What is a "period" ? Is it a month, or something more "custom" ? Can
periods overlap ?

> COALESCE(( SELECT sum(gltrans.gltrans_amount) AS sum
> FROM gltrans
> WHERE gltrans.gltrans_date < period.period_start
> AND gltrans.gltrans_accnt_id = accnt.accnt_id
> AND gltrans.gltrans_posted = true), 0.00)::text::money AS
> beginbalance,


Note that here you are scanning the entire table multiple times, the
complexity of this is basically (rows in gltrans)^2 which is something
you'd like to avoid.

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #7 (permalink)  
Old 05-07-2008, 10:18 AM
Justin
 
Posts: n/a
Default Re: need to speed up query



PFC wrote:
>
>> i've had to write queries to get trail balance values out of the GL
>> transaction table and i'm not happy with its performance The table
>> has 76K rows growing about 1000 rows per working day so the
>> performance is not that great it takes about 20 to 30 seconds to get
>> all the records for the table and when we limit it to single
>> accounting period it drops down to 2 seconds

>
> What is a "period" ? Is it a month, or something more "custom" ?
> Can periods overlap ?

No periods can never overlap. If the periods did you would be in
violation of many tax laws around the world. Plus it you would not know
how much money you are making or losing.
Generally yes a accounting period is a normal calendar month. but you
can have 13 periods in a normal calendar year. 52 weeks in a year / 4
weeks in month = 13 periods or 13 months in a Fiscal Calendar year.
This means if someone is using a 13 period fiscal accounting year the
start and end dates are offset from a normal calendar.
To make this really funky you can have a Fiscal Calendar year start
June 15 2008 and end on June 14 2009

http://en.wikipedia.org/wiki/Fiscal_year
>
>> COALESCE(( SELECT sum(gltrans.gltrans_amount) AS sum
>> FROM gltrans
>> WHERE gltrans.gltrans_date < period.period_start
>> AND gltrans.gltrans_accnt_id = accnt.accnt_id
>> AND gltrans.gltrans_posted = true), 0.00)::text::money AS
>> beginbalance,

>
> Note that here you are scanning the entire table multiple times,
> the complexity of this is basically (rows in gltrans)^2 which is
> something you'd like to avoid.
>

For accounting purposes you need to know the Beginning Balances,
Debits, Credits, Difference between Debits to Credits and the Ending
Balance for each account. We have 133 accounts with presently 12
periods defined so we end up 1596 rows returned for this query.

So period 1 should have for the most part have Zero for Beginning
Balances for most types of Accounts. Period 2 is Beginning Balance is
Period 1 Ending Balance, Period 3 is Period 2 ending balance so and so
on forever.






--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #8 (permalink)  
Old 05-07-2008, 10:19 AM
PFC
 
Posts: n/a
Default Re: need to speed up query


>> What is a "period" ? Is it a month, or something more "custom" ?
>> Can periods overlap ?


> No periods can never overlap. If the periods did you would be in
> violation of many tax laws around the world. Plus it you would not know
> how much money you are making or losing.


I was wondering if you'd be using the same query to compute how much was
gained every month and every week, which would have complicated things.
But now it's clear.

> To make this really funky you can have a Fiscal Calendar year start
> June 15 2008 and end on June 14 2009


Don't you just love those guys ? Always trying new tricks to make your
life more interesting.

>> Note that here you are scanning the entire table multiple times,
>> the complexity of this is basically (rows in gltrans)^2 which is
>> something you'd like to avoid.
>>

> For accounting purposes you need to know the Beginning Balances,
> Debits, Credits, Difference between Debits to Credits and the Ending
> Balance for each account. We have 133 accounts with presently 12
> periods defined so we end up 1596 rows returned for this query.


Alright, I propose a solution which only works when periods don't overlap.
It will scan the entire table, but only once, not many times as your
current query does.

> So period 1 should have for the most part have Zero for Beginning
> Balances for most types of Accounts. Period 2 is Beginning Balance is
> Period 1 Ending Balance, Period 3 is Period 2 ending balance so and so
> on forever.


Precisely. So, it is not necessary to recompute everything for each
period.
Use the previous period's ending balance as the current period's starting
balance...

There are several ways to do this.
First, you could use your current query, but only compute the sum of what
happened during a period, for each period, and store that in a temporary
table.
Then, you use a plpgsql function, or you do that in your client, you take
the rows in chronological order, you sum them as they come, and you get
your balances. Use a NUMERIC type, not a FLOAT, to avoid rounding errors.

The other solution does the same thing but optimizes the first step like
this :
INSERT INTO temp_table SELECT period, sum(...) GROUP BY period

To do this you must be able to compute the period from the date and not
the other way around. You could store a period_id in your table, or use a
function.

Another much more efficient solution would be to have a summary table
which keeps the summary data for each period, with beginning balance and
end balance. This table will only need to be updated when someone finds an
old receipt in their pocket or something.

> This falls under the stupid question and i'm just curious what other
> people think what makes a query complex?


I have some rather complex queries which postgres burns in a few
milliseconds.
You could define complexity as the amount of brain sweat that went into
writing that query.
You could also define complexity as O(n) or O(n^2) etc, for instance your
query (as written) is O(n^2) which is something you don't want, I've seen
stuff that was O(2^n) or worse, O(n!) in software written by drunk
students, in this case getting rid of it is an emergency...

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #9 (permalink)  
Old 05-07-2008, 10:19 AM
Shaun Thomas
 
Posts: n/a
Default Re: need to speed up query

On Tue, 2008-05-06 at 03:01 +0100, Justin wrote:

> i've had to write queries to get trail balance values out of the GL
> transaction table and i'm not happy with its performance


Go ahead and give this a try:

SELECT p.period_id, p.period_start, p.period_end, a.accnt_id,
a.accnt_number, a.accnt_descrip, p.period_yearperiod_id,
a.accnt_type,
SUM(CASE WHEN g.gltrans_date < p.period_start
THEN g.gltrans_amount ELSE 0.0
END)::text::money AS beginbalance,
SUM(CASE WHEN g.gltrans_date < p.period_end
AND g.gltrans_date >= p.period_start
AND g.gltrans_amount <= 0::numeric
THEN g.gltrans_amount ELSE 0.0
END)::text::money AS negative,
SUM(CASE WHEN g.gltrans_date <= p.period_end
AND g.gltrans_date >= p.period_start
AND g.gltrans_amount >= 0::numeric
THEN g.gltrans_amount ELSE 0.0
END)::text::money AS positive,
SUM(CASE WHEN g.gltrans_date <= p.period_end
AND g.gltrans_date >= p.period_start
THEN g.gltrans_amount ELSE 0.0
END)::text::money AS difference,
SUM(CASE WHEN g.gltrans_date <= p.period_end
THEN g.gltrans_amount ELSE 0.0
END)::text::money AS endbalance,
FROM period p
CROSS JOIN accnt a
LEFT JOIN gltrans g ON (g.gltrans_accnt_id = a.accnt_id
AND g.gltrans_posted = true)
ORDER BY period.period_id, accnt.accnt_number;

Depending on how the planner saw your old query, it may have forced
several different sequence or index scans to get the information from
gltrans. One thing all of your subqueries had in common was a join on
the account id and listing only posted transactions. It's still a big
gulp, but it's only one gulp.

The other thing I did was that I guessed you added the coalesce clause
because the subqueries individually could return null rowsets for
various groupings, and you wouldn't want that. This left-join solution
only lets it add to your various sums if it matches all the conditions,
otherwise it falls through the list of cases until nothing matches. If
some of your transactions can have null amounts, you might consider
turning g.gltrans into COALESCE(g.gltrans, 0.0) instead.

Otherwise, this *might* work; without knowing more about your schema,
it's only a guess. I'm a little skeptical about the conditionless
cross-join, but whatever.

Either way, by looking at this query, it looks like some year-end
summary piece, or an at-a-glance idea of your account standings. The
problem you're going to have with this is that there's no way to truly
optimize this. One way or another, you're going to incur some
combination of three sequence scans or three index scans; if those
tables get huge, you're in trouble. You might want to consider a
denormalized summary table that contains this information (and maybe
more) maintained by a trigger or regularly invoked stored-procedure and
then you can select from *that* with much less agony.

Then there's fact-tables, but that's beyond the scope of this email.

Good luck!

--

Shaun Thomas
Database Administrator

Leapfrog Online
807 Greenwood Street
Evanston, IL 60201
Tel. 847-440-8253
Fax. 847-570-5750
www.leapfrogonline.com



--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #10 (permalink)  
Old 05-07-2008, 10:19 AM
Justin
 
Posts: n/a
Default Re: need to speed up query

it worked it had couple missing parts but it worked and ran in 3.3
seconds. *Thanks for this *
i need to review the result and balance it to my results as the
Accountant already went through and balanced some accounts by hand to
verify my results

<<begin quote>>

You might want to consider a
denormalized summary table that contains this information (and maybe
more) maintained by a trigger or regularly invoked stored-procedure and
then you can select from *that* with much less agony.

<<end quote>>

I just dumped the summary table because it kept getting out of balance
all the time and was missing accounts that did not have transaction in
them for given period. Again i did not lay out the table nor the old
code which was terrible and did not work correctly. I tried several
times to fix the summary table but to many things allowed it to get
out of sync. Keeping the Ending and Beginning Balance correct was to
much trouble and i needed to get numbers we can trust to the accountant.

The developers of the code got credits and debits backwards so instead
of fixing the code they just added code to flip the values on the front
end. Its really annoying. At this point if i could go back 7 months
ago i would not purchased this software if i had known what i know now.

I've had to make all kinds of changes i never intended to make in order
to get the stuff to balance and agree. I've spent the last 3 months in
code review fixing things that allow accounts to get out of balance and
stop stupid things from happening, like posting GL Transactions into
non-existing accounting periods. the list of things i have to fix is
getting dam long.



Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT. The time now is 11:06 AM.


Powered by vBulletin® Version 3.6.5
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
Search Engine Optimization by vBSEO 3.1.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538