SEO

#1 (**permalink**) 02-16-2008, 06:45 AM

I have an application running on OBSD 3.5 on an i386 machine with 4GB. It's very
memory-hungry (builds a big cache of retrieved on-disk data) and would happily
use all available memory (which I'd like) - unfortunately, it is limited by
RLIMIT_DATA to 1048576 KB (it allocates memory from the heap).

I am investigating using an mmap'd file for the buffer, but would also like to
understand if I can increase the hard limit of RLIMIT_DATA. Is this being set by
the MAXDSIZ kernel parameter? Can I "just" raise that number in vmparam.h and
recompile, or are there some fundamental limits based on structure sizes?

Thanks.

#2 (**permalink**) 02-16-2008, 06:45 AM

Chris Kantarjiev wrote:

> I am investigating using an mmap'd file for the buffer, but would also like to
> understand if I can increase the hard limit of RLIMIT_DATA. Is this being set by
> the MAXDSIZ kernel parameter? Can I "just" raise that number in vmparam.h and
> recompile, or are there some fundamental limits based on structure sizes?

For all released versions, you can use mmap (with MAP_ANON, so you
don't need a file either) to get around the limit. You won't be able
to allocate it all at once, but with 2 or 3 calls you can come close.
Changing anything in vmparam.h is a very bad idea.

#3 (**permalink**) 02-16-2008, 06:46 AM

tedu wrote:
>
> Chris Kantarjiev wrote:
>
>
>>I am investigating using an mmap'd file for the buffer, but would also like to
>>understand if I can increase the hard limit of RLIMIT_DATA. Is this being set by
>>the MAXDSIZ kernel parameter? Can I "just" raise that number in vmparam.h and
>>recompile, or are there some fundamental limits based on structure sizes?
>
>
> For all released versions, you can use mmap (with MAP_ANON, so you
> don't need a file either) to get around the limit. You won't be able
> to allocate it all at once, but with 2 or 3 calls you can come close.
> Changing anything in vmparam.h is a very bad idea.
>

Using mmap() would be a great idea if I was working in C. I'm not; I'm working
in python, so the code that allocates the arena is several layers down. Changing
it to use mmap() is not something I'm going to do in my spare time over the next
week.

So. Care to expound on *why* changing MAXDSIZ is a 'very bad idea'? I'd be happy
just to double it (to 2GB) which would still make certain that the kernel has
all the room it can want.

#4 (**permalink**) 02-16-2008, 06:46 AM

for the long answer, grep MAXDSIZ in the kernel. basically, it sets
the size of the heap, which can't be used for other things like mmap().
when this came up two weeks ago, the guy tried it and later told me
"yeah, it didn't work too well". your experience may be different, but
you're on your own.
http://marc.theaimsgroup.com/?t=111731093400001&r=1&w=2

#5 (**permalink**) 02-16-2008, 06:46 AM

tedu wrote:
> for the long answer, grep MAXDSIZ in the kernel. basically, it sets
> the size of the heap, which can't be used for other things like mmap().
> when this came up two weeks ago, the guy tried it and later told me
> "yeah, it didn't work too well". your experience may be different, but
> you're on your own.
> http://marc.theaimsgroup.com/?t=111731093400001&r=1&w=2
>

OK, I'll take that as an answer, thanks.

I took a different tack. Cursory reading of malloc.c led me to believe that it
uses mmap to allocate its pages if they're pagesize or more. I wrote a simple
program and found that wasn't actually true - despite the fact that there's an
internal function called "map_pages", it uses brk() to get those pages.

Perhaps a future edition of malloc() could take an option use mmap() to those
pages...

Thanks,
chris

#6 (**permalink**) 02-16-2008, 06:46 AM

For grins, I compiled my little program with the gnu malloc, which never calls
sbrk anywhere. It's slightly better than the 1023 MB I was getting before:

dick 331 % limit
cputime unlimited
filesize unlimited
datasize 1048576 kbytes
stacksize 4096 kbytes
coredumpsize 1000 kbytes
memoryuse 3642208 kbytes
memorylocked 3643304 kbytes
maxproc 128
openfiles 64
dick 332 % glutton
Cannot map anonymous memorychunk: 1 MB, final size: 1284096 KB 1254 MB

I wonder what limit I'm hitting now? The machine has 4GB physical.

#7 (**permalink**) 02-16-2008, 06:46 AM

And, if anyone wants to experiment, here's my little program:

dick 340 % cat glutton.c
#include <stdio.h>
#include <stdlib.h>

#define KB (1024)
#define MB (1024*KB)
#define CHUNK 1*MB
main()
{
void *p;
long int size = 0;

do {
p = malloc((size_t) CHUNK);
if (p != NULL)
size += CHUNK;
} while (p != NULL);
printf("chunk: %d MB, final size: %d KB %d MB\n",
CHUNK/MB,
size/KB,
size/MB);
}

#8 (**permalink**) 02-16-2008, 06:46 AM

have a look at the new malloc.c in cvs. it uses mmap.
(the kernel has since changed to account mmap as part of your data
limit, but not in 3.7)

#9 (**permalink**) 02-16-2008, 06:46 AM

tedu wrote:
> have a look at the new malloc.c in cvs. it uses mmap.
> (the kernel has since changed to account mmap as part of your data
> limit, but not in 3.7)
>

OK, I grabbed that and compiled. I'm still coming up against a limit of 1284096
KB 1254 MB despite

dick 359 % limit
cputime unlimited
filesize unlimited
datasize 1048576 kbytes
stacksize 4096 kbytes
coredumpsize 1000 kbytes
memoryuse 3642208 kbytes
memorylocked 3643304 kbytes
maxproc 128
openfiles 64

This is a 3.5 machine... any ideas?

Thanks again.

#10 (**permalink**) 02-16-2008, 06:46 AM

"tedu" <tu@zeitbombe.org> writes:
> have a look at the new malloc.c in cvs. it uses mmap.
> (the kernel has since changed to account mmap as part of your data
> limit, but not in 3.7)

Mmap and/or the page table code has also changed for the better since
the last time I tried my mmap test. I can now mmap 127.50 TB .

Now I'm not brave enough to try to read past the end of my mmapped
file. I'm assuming it allows the mmap to extend past the end of the
underlying file but will hit one with a SIGBUS if one actually tried
to access it?

-wolfgang
--
Wolfgang S. Rupprecht http://www.wsrcc.com/wolfgang/

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

SEO

changing MAXDSIZ/using more physical memory?