vBulletin Search Engine Optimization
| |||||||
| Register | FAQ | Members List | Calendar | Search | Today's Posts | Mark Forums Read |
| ||||
| Hello, I have noticed a strange performance regression and I'm at a loss as to what's happening. We have a fairly large database (~16 GB). The original postgres 7.4 was running on a sun v880 with 4 CPUs and 8 GB of ram running Solaris on local scsi discs. The new server is a sun Opteron box with 4 cores, 8 GB of ram running postgres 8.1.4 on Linux (AMD64) on a 4 Gbps FC SAN volume. When we created the new database it was created from scratch rather than copying over the old one, however the table structure is almost identical (UTF8 on the new one vs. C on the old). The problem is queries are ~10x slower on the new hardware. I read several places that the SAN might be to blame, but testing with bonnie and dd indicates that the SAN is actually almost twice as fast as the scsi discs in the old sun server. I've tried adjusting just about every option in the postgres config file, but performance remains the same. Any ideas? Thanks, Alex ---------------------------(end of broadcast)--------------------------- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq |
| |||
| Alex Deucher wrote: > Hello, > > I have noticed a strange performance regression and I'm at a loss as > to what's happening. We have a fairly large database (~16 GB). The > original postgres 7.4 was running on a sun v880 with 4 CPUs and 8 GB > of ram running Solaris on local scsi discs. The new server is a sun > Opteron box with 4 cores, 8 GB of ram running postgres 8.1.4 on Linux > (AMD64) on a 4 Gbps FC SAN volume. When we created the new database > it was created from scratch rather than copying over the old one, > however the table structure is almost identical (UTF8 on the new one > vs. C on the old). The problem is queries are ~10x slower on the new > hardware. I read several places that the SAN might be to blame, but > testing with bonnie and dd indicates that the SAN is actually almost > twice as fast as the scsi discs in the old sun server. I've tried > adjusting just about every option in the postgres config file, but > performance remains the same. Any ideas? Vacuum? Analayze? default_statistics_target? How many shared_buffers? effective_cache_size? work_mem? Sincerely, Joshua D. Drake > > Thanks, > > Alex > > ---------------------------(end of broadcast)--------------------------- > TIP 3: Have you checked our extensive FAQ? > > http://www.postgresql.org/docs/faq > -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 Providing the most comprehensive PostgreSQL solutions since 1997 http://www.commandprompt.com/ Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ ---------------------------(end of broadcast)--------------------------- TIP 4: Have you searched our list archives? http://archives.postgresql.org |
| |||
| On Thu, 1 Mar 2007, Joshua D. Drake wrote: > Alex Deucher wrote: >> Hello, >> >> I have noticed a strange performance regression and I'm at a loss as >> to what's happening. We have a fairly large database (~16 GB). The >> original postgres 7.4 was running on a sun v880 with 4 CPUs and 8 GB >> of ram running Solaris on local scsi discs. The new server is a sun >> Opteron box with 4 cores, 8 GB of ram running postgres 8.1.4 on Linux >> (AMD64) on a 4 Gbps FC SAN volume. When we created the new database >> it was created from scratch rather than copying over the old one, >> however the table structure is almost identical (UTF8 on the new one >> vs. C on the old). The problem is queries are ~10x slower on the new >> hardware. I read several places that the SAN might be to blame, but >> testing with bonnie and dd indicates that the SAN is actually almost >> twice as fast as the scsi discs in the old sun server. I've tried >> adjusting just about every option in the postgres config file, but >> performance remains the same. Any ideas? > > Vacuum? Analayze? default_statistics_target? How many shared_buffers? > effective_cache_size? work_mem? Also, an explain analyze from both the 7.4 and 8.1 systems with one of the 10x slower queries would probably be handy. What do you mean by "created from scratch rather than copying over the old one"? How did you put the data in? Did you run analyze after loading it? Is autovacuum enabled and if so, what are the thresholds? -- Jeff Frost, Owner <jeff@frostconsultingllc.com> Frost Consulting, LLC http://www.frostconsultingllc.com/ Phone: 650-780-7908 FAX: 650-649-1954 ---------------------------(end of broadcast)--------------------------- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate |
| |||
| On 3/1/07, Joshua D. Drake <jd@commandprompt.com> wrote: > Alex Deucher wrote: > > Hello, > > > > I have noticed a strange performance regression and I'm at a loss as > > to what's happening. We have a fairly large database (~16 GB). The > > original postgres 7.4 was running on a sun v880 with 4 CPUs and 8 GB > > of ram running Solaris on local scsi discs. The new server is a sun > > Opteron box with 4 cores, 8 GB of ram running postgres 8.1.4 on Linux > > (AMD64) on a 4 Gbps FC SAN volume. When we created the new database > > it was created from scratch rather than copying over the old one, > > however the table structure is almost identical (UTF8 on the new one > > vs. C on the old). The problem is queries are ~10x slower on the new > > hardware. I read several places that the SAN might be to blame, but > > testing with bonnie and dd indicates that the SAN is actually almost > > twice as fast as the scsi discs in the old sun server. I've tried > > adjusting just about every option in the postgres config file, but > > performance remains the same. Any ideas? > > Vacuum? Analayze? default_statistics_target? How many shared_buffers? > effective_cache_size? work_mem? > I'm running the autovacuum process on the 8.1 server. vacuuming on the old server was done manually. default_statistics_target and effective_cache_size are set to the the defaults on both. postgres 7.4 server: # - Memory - shared_buffers = 82000 # 1000 min 16, at least max_connections*2, 8KB each sort_mem = 8000 # 1024 min 64, size in KB vacuum_mem = 32000 # 8192 min 1024, size in KB # - Free Space Map - #max_fsm_pages = 20000 # min max_fsm_relations*16, 6 bytes each #max_fsm_relations = 1000 # min 100, ~50 bytes each # - Kernel Resource Usage - #max_files_per_process = 1000 # min 25 postgres 8.1 server: # - Memory - shared_buffers = 100000 # min 16 or max_connections*2, 8KB each temp_buffers = 2000 #1000 # min 100, 8KB each max_prepared_transactions = 100 #5 # can be 0 or more # note: increasing max_prepared_transactions costs ~600 bytes of shared memory # per transaction slot, plus lock space (see max_locks_per_transaction). work_mem = 10000 #1024 # min 64, size in KB maintenance_work_mem = 524288 #16384 # min 1024, size in KB #max_stack_depth = 2048 # min 100, size in KB I've also tried using the same settings from the old server on the new one; same performance issues. Thanks, Alex > Sincerely, > > Joshua D. Drake > > > > > > Thanks, > > > > Alex > > > > ---------------------------(end of broadcast)--------------------------- > > TIP 3: Have you checked our extensive FAQ? > > > > http://www.postgresql.org/docs/faq > > > > > -- > > === The PostgreSQL Company: Command Prompt, Inc. === > Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 > Providing the most comprehensive PostgreSQL solutions since 1997 > http://www.commandprompt.com/ > > Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate > PostgreSQL Replication: http://www.commandprompt.com/products/ > > ---------------------------(end of broadcast)--------------------------- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq |
| |||
| On 3/1/07, Jeff Frost <jeff@frostconsultingllc.com> wrote: > On Thu, 1 Mar 2007, Joshua D. Drake wrote: > > > Alex Deucher wrote: > >> Hello, > >> > >> I have noticed a strange performance regression and I'm at a loss as > >> to what's happening. We have a fairly large database (~16 GB). The > >> original postgres 7.4 was running on a sun v880 with 4 CPUs and 8 GB > >> of ram running Solaris on local scsi discs. The new server is a sun > >> Opteron box with 4 cores, 8 GB of ram running postgres 8.1.4 on Linux > >> (AMD64) on a 4 Gbps FC SAN volume. When we created the new database > >> it was created from scratch rather than copying over the old one, > >> however the table structure is almost identical (UTF8 on the new one > >> vs. C on the old). The problem is queries are ~10x slower on the new > >> hardware. I read several places that the SAN might be to blame, but > >> testing with bonnie and dd indicates that the SAN is actually almost > >> twice as fast as the scsi discs in the old sun server. I've tried > >> adjusting just about every option in the postgres config file, but > >> performance remains the same. Any ideas? > > > > Vacuum? Analayze? default_statistics_target? How many shared_buffers? > > effective_cache_size? work_mem? > > Also, an explain analyze from both the 7.4 and 8.1 systems with one of the > 10x slower queries would probably be handy. > I'll run some and get back to you. > What do you mean by "created from scratch rather than copying over the old > one"? How did you put the data in? Did you run analyze after loading it? > Is autovacuum enabled and if so, what are the thresholds? Both the databases were originally created from xml files. We just re-created the new one from the xml rather than copying the old database over. I didn't manually run analyze on it, but we are running the autovacuum process: autovacuum = on #off # enable autovacuum subprocess? autovacuum_naptime = 360 #60 # time between autovacuum runs, in secs autovacuum_vacuum_threshold = 10000 #1000 # min # of tuple updates before # vacuum autovacuum_analyze_threshold = 5000 #500 # min # of tuple updates before Thanks, Alex > > -- > Jeff Frost, Owner <jeff@frostconsultingllc.com> > Frost Consulting, LLC http://www.frostconsultingllc.com/ > Phone: 650-780-7908 FAX: 650-649-1954 > ---------------------------(end of broadcast)--------------------------- TIP 2: Don't 'kill -9' the postmaster |
| |||
| On Thu, 1 Mar 2007, Alex Deucher wrote: > On 3/1/07, Jeff Frost <jeff@frostconsultingllc.com> wrote: >> On Thu, 1 Mar 2007, Joshua D. Drake wrote: >> >> > Alex Deucher wrote: >> >> Hello, >> >> >> >> I have noticed a strange performance regression and I'm at a loss as >> >> to what's happening. We have a fairly large database (~16 GB). The >> >> original postgres 7.4 was running on a sun v880 with 4 CPUs and 8 GB >> >> of ram running Solaris on local scsi discs. The new server is a sun >> >> Opteron box with 4 cores, 8 GB of ram running postgres 8.1.4 on Linux >> >> (AMD64) on a 4 Gbps FC SAN volume. When we created the new database >> >> it was created from scratch rather than copying over the old one, >> >> however the table structure is almost identical (UTF8 on the new one >> >> vs. C on the old). The problem is queries are ~10x slower on the new >> >> hardware. I read several places that the SAN might be to blame, but >> >> testing with bonnie and dd indicates that the SAN is actually almost >> >> twice as fast as the scsi discs in the old sun server. I've tried >> >> adjusting just about every option in the postgres config file, but >> >> performance remains the same. Any ideas? >> > >> > Vacuum? Analayze? default_statistics_target? How many shared_buffers? >> > effective_cache_size? work_mem? >> >> Also, an explain analyze from both the 7.4 and 8.1 systems with one of the >> 10x slower queries would probably be handy. >> > > I'll run some and get back to you. > >> What do you mean by "created from scratch rather than copying over the old >> one"? How did you put the data in? Did you run analyze after loading it? >> Is autovacuum enabled and if so, what are the thresholds? > > Both the databases were originally created from xml files. We just > re-created the new one from the xml rather than copying the old > database over. I didn't manually run analyze on it, but we are > running the autovacuum process: You should probably manually run analyze and see if that resolves your problem. > > autovacuum = on #off # enable autovacuum subprocess? > autovacuum_naptime = 360 #60 # time between autovacuum runs, in > secs > autovacuum_vacuum_threshold = 10000 #1000 # min # of tuple updates > before > # vacuum > autovacuum_analyze_threshold = 5000 #500 # min # of tuple updates > before Most people make autovacuum more aggressive and not less aggressive. In fact, the new defaults in 8.2 are: #autovacuum_vacuum_threshold = 500 # min # of tuple updates before # vacuum #autovacuum_analyze_threshold = 250 # min # of tuple updates before # analyze #autovacuum_vacuum_scale_factor = 0.2 # fraction of rel size before # vacuum #autovacuum_analyze_scale_factor = 0.1 # fraction of rel size before I'd recommend trying those, otherwise you might not vacuum enough. It'll be interesting to see the explain analyze output after you've run analyze by hand. -- Jeff Frost, Owner <jeff@frostconsultingllc.com> Frost Consulting, LLC http://www.frostconsultingllc.com/ Phone: 650-780-7908 FAX: 650-649-1954 ---------------------------(end of broadcast)--------------------------- TIP 6: explain analyze is your friend |
| |||
| On Thu, 1 Mar 2007, Alex Deucher wrote: >> Vacuum? Analayze? default_statistics_target? How many shared_buffers? >> effective_cache_size? work_mem? >> > > I'm running the autovacuum process on the 8.1 server. vacuuming on > the old server was done manually. > > default_statistics_target and effective_cache_size are set to the the > defaults on both. > > postgres 7.4 server: > # - Memory - > shared_buffers = 82000 # 1000 min 16, at least > max_connections*2, 8KB each > sort_mem = 8000 # 1024 min 64, size in KB > vacuum_mem = 32000 # 8192 min 1024, size in KB > # - Free Space Map - > #max_fsm_pages = 20000 # min max_fsm_relations*16, 6 bytes each > #max_fsm_relations = 1000 # min 100, ~50 bytes each > # - Kernel Resource Usage - > #max_files_per_process = 1000 # min 25 > > postgres 8.1 server: > # - Memory - > shared_buffers = 100000 # min 16 or max_connections*2, 8KB > each > temp_buffers = 2000 #1000 # min 100, 8KB each > max_prepared_transactions = 100 #5 # can be 0 or more > # note: increasing max_prepared_transactions costs ~600 bytes of shared > memory > # per transaction slot, plus lock space (see max_locks_per_transaction). > work_mem = 10000 #1024 # min 64, size in KB > maintenance_work_mem = 524288 #16384 # min 1024, size in KB > #max_stack_depth = 2048 # min 100, size in KB > > I've also tried using the same settings from the old server on the new > one; same performance issues. > If this is a linux system, could you give us the output of the 'free' command? Postgresql might be choosing a bad plan because your effective_cache_size is way off (it's the default now right?). Also, what was the block read/write speed of the SAN from your bonnie tests? Probably want to tune random_page_cost as well if it's also at the default. -- Jeff Frost, Owner <jeff@frostconsultingllc.com> Frost Consulting, LLC http://www.frostconsultingllc.com/ Phone: 650-780-7908 FAX: 650-649-1954 ---------------------------(end of broadcast)--------------------------- TIP 5: don't forget to increase your free space map settings |
| |||
| On 3/1/07, Jeff Frost <jeff@frostconsultingllc.com> wrote: > On Thu, 1 Mar 2007, Alex Deucher wrote: > > >> Vacuum? Analayze? default_statistics_target? How many shared_buffers? > >> effective_cache_size? work_mem? > >> > > > > I'm running the autovacuum process on the 8.1 server. vacuuming on > > the old server was done manually. > > > > default_statistics_target and effective_cache_size are set to the the > > defaults on both. > > > > postgres 7.4 server: > > # - Memory - > > shared_buffers = 82000 # 1000 min 16, at least > > max_connections*2, 8KB each > > sort_mem = 8000 # 1024 min 64, size in KB > > vacuum_mem = 32000 # 8192 min 1024, size in KB > > # - Free Space Map - > > #max_fsm_pages = 20000 # min max_fsm_relations*16, 6 bytes each > > #max_fsm_relations = 1000 # min 100, ~50 bytes each > > # - Kernel Resource Usage - > > #max_files_per_process = 1000 # min 25 > > > > postgres 8.1 server: > > # - Memory - > > shared_buffers = 100000 # min 16 or max_connections*2, 8KB > > each > > temp_buffers = 2000 #1000 # min 100, 8KB each > > max_prepared_transactions = 100 #5 # can be 0 or more > > # note: increasing max_prepared_transactions costs ~600 bytes of shared > > memory > > # per transaction slot, plus lock space (see max_locks_per_transaction). > > work_mem = 10000 #1024 # min 64, size in KB > > maintenance_work_mem = 524288 #16384 # min 1024, size in KB > > #max_stack_depth = 2048 # min 100, size in KB > > > > I've also tried using the same settings from the old server on the new > > one; same performance issues. > > > > If this is a linux system, could you give us the output of the 'free' command? total used free shared buffers cached Mem: 8059852 8042868 16984 0 228 7888648 -/+ buffers/cache: 153992 7905860 Swap: 15631224 2164 15629060 > Postgresql might be choosing a bad plan because your effective_cache_size is > way off (it's the default now right?). Also, what was the block read/write yes it's set to the default. > speed of the SAN from your bonnie tests? Probably want to tune > random_page_cost as well if it's also at the default. > ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP luna12-san 16000M 58896 91 62931 9 35870 5 54869 82 145504 13 397.7 0 effective_cache_size is the default. Alex ---------------------------(end of broadcast)--------------------------- TIP 6: explain analyze is your friend |
| |||
| On Thu, 1 Mar 2007, Alex Deucher wrote: > On 3/1/07, Jeff Frost <jeff@frostconsultingllc.com> wrote: >> On Thu, 1 Mar 2007, Alex Deucher wrote: >> >> >> Vacuum? Analayze? default_statistics_target? How many shared_buffers? >> >> effective_cache_size? work_mem? >> >> >> > >> > I'm running the autovacuum process on the 8.1 server. vacuuming on >> > the old server was done manually. >> > >> > default_statistics_target and effective_cache_size are set to the the >> > defaults on both. >> > >> > postgres 7.4 server: >> > # - Memory - >> > shared_buffers = 82000 # 1000 min 16, at least >> > max_connections*2, 8KB each >> > sort_mem = 8000 # 1024 min 64, size in KB >> > vacuum_mem = 32000 # 8192 min 1024, size in KB >> > # - Free Space Map - >> > #max_fsm_pages = 20000 # min max_fsm_relations*16, 6 bytes each >> > #max_fsm_relations = 1000 # min 100, ~50 bytes each >> > # - Kernel Resource Usage - >> > #max_files_per_process = 1000 # min 25 >> > >> > postgres 8.1 server: >> > # - Memory - >> > shared_buffers = 100000 # min 16 or max_connections*2, >> 8KB >> > each >> > temp_buffers = 2000 #1000 # min 100, 8KB each >> > max_prepared_transactions = 100 #5 # can be 0 or more >> > # note: increasing max_prepared_transactions costs ~600 bytes of shared >> > memory >> > # per transaction slot, plus lock space (see max_locks_per_transaction). >> > work_mem = 10000 #1024 # min 64, size in KB >> > maintenance_work_mem = 524288 #16384 # min 1024, size in KB >> > #max_stack_depth = 2048 # min 100, size in KB >> > >> > I've also tried using the same settings from the old server on the new >> > one; same performance issues. >> > >> >> If this is a linux system, could you give us the output of the 'free' >> command? > > total used free shared buffers cached > Mem: 8059852 8042868 16984 0 228 7888648 > -/+ buffers/cache: 153992 7905860 > Swap: 15631224 2164 15629060 So, I would set effective_cache_size = 988232 (7905860/8). > >> Postgresql might be choosing a bad plan because your effective_cache_size >> is >> way off (it's the default now right?). Also, what was the block read/write > > yes it's set to the default. > >> speed of the SAN from your bonnie tests? Probably want to tune >> random_page_cost as well if it's also at the default. >> > > ------Sequential Output------ --Sequential Input- > --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec > %CP > luna12-san 16000M 58896 91 62931 9 35870 5 54869 82 145504 13 397.7 > 0 > So, you're getting 62MB/s writes and 145MB/s reads. Just FYI, that write speed is about the same as my single SATA drive write speed on my workstation, so not that great. The read speed is decent, though and with that sort of read performance, you might want to lower random_page_cost to something like 2.5 or 2 so the planner will tend to prefer index scans. -- Jeff Frost, Owner <jeff@frostconsultingllc.com> Frost Consulting, LLC http://www.frostconsultingllc.com/ Phone: 650-780-7908 FAX: 650-649-1954 ---------------------------(end of broadcast)--------------------------- TIP 6: explain analyze is your friend |
| ||||
| On 3/1/07, Jeff Frost <jeff@frostconsultingllc.com> wrote: > On Thu, 1 Mar 2007, Alex Deucher wrote: > > > On 3/1/07, Jeff Frost <jeff@frostconsultingllc.com> wrote: > >> On Thu, 1 Mar 2007, Alex Deucher wrote: > >> > >> >> Vacuum? Analayze? default_statistics_target? How many shared_buffers? > >> >> effective_cache_size? work_mem? > >> >> > >> > > >> > I'm running the autovacuum process on the 8.1 server. vacuuming on > >> > the old server was done manually. > >> > > >> > default_statistics_target and effective_cache_size are set to the the > >> > defaults on both. > >> > > >> > postgres 7.4 server: > >> > # - Memory - > >> > shared_buffers = 82000 # 1000 min 16, at least > >> > max_connections*2, 8KB each > >> > sort_mem = 8000 # 1024 min 64, size in KB > >> > vacuum_mem = 32000 # 8192 min 1024, size in KB > >> > # - Free Space Map - > >> > #max_fsm_pages = 20000 # min max_fsm_relations*16, 6 bytes each > >> > #max_fsm_relations = 1000 # min 100, ~50 bytes each > >> > # - Kernel Resource Usage - > >> > #max_files_per_process = 1000 # min 25 > >> > > >> > postgres 8.1 server: > >> > # - Memory - > >> > shared_buffers = 100000 # min 16 or max_connections*2, > >> 8KB > >> > each > >> > temp_buffers = 2000 #1000 # min 100, 8KB each > >> > max_prepared_transactions = 100 #5 # can be 0 or more > >> > # note: increasing max_prepared_transactions costs ~600 bytes of shared > >> > memory > >> > # per transaction slot, plus lock space (see max_locks_per_transaction). > >> > work_mem = 10000 #1024 # min 64, size in KB > >> > maintenance_work_mem = 524288 #16384 # min 1024, size in KB > >> > #max_stack_depth = 2048 # min 100, size in KB > >> > > >> > I've also tried using the same settings from the old server on the new > >> > one; same performance issues. > >> > > >> > >> If this is a linux system, could you give us the output of the 'free' > >> command? > > > > total used free shared buffers cached > > Mem: 8059852 8042868 16984 0 228 7888648 > > -/+ buffers/cache: 153992 7905860 > > Swap: 15631224 2164 15629060 > > So, I would set effective_cache_size = 988232 (7905860/8). > > > > >> Postgresql might be choosing a bad plan because your effective_cache_size > >> is > >> way off (it's the default now right?). Also, what was the block read/write > > > > yes it's set to the default. > > > >> speed of the SAN from your bonnie tests? Probably want to tune > >> random_page_cost as well if it's also at the default. > >> > > > > ------Sequential Output------ --Sequential Input- > > --Random- > > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > > --Seeks-- > > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec > > %CP > > luna12-san 16000M 58896 91 62931 9 35870 5 54869 82 145504 13 397.7 > > 0 > > > > So, you're getting 62MB/s writes and 145MB/s reads. Just FYI, that write > speed is about the same as my single SATA drive write speed on my workstation, > so not that great. The read speed is decent, though and with that sort of > read performance, you might want to lower random_page_cost to something like > 2.5 or 2 so the planner will tend to prefer index scans. > Right, but the old box was getting ~45MBps on both reads and writes, so it's an improvement for me know how it goes. Alex ---------------------------(end of broadcast)--------------------------- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq |