Hello,

I'm having trouble importing a LDIF file into an IPlanet Directory
Server 4.16 (IPDS) on Solaris 8.
The LDIF is large, it's size is 2.4 GB, so I suspect that the problem
is the inability of IPDS to open files larger than 2 GB.

I want to confirm that there is a limitation of the software to open
large files, before to get into the effort to split the file in
smaller pieces.

If so, is there some way to avoid the limitation?...

Any help will be greatly appreciated.

Thank you in advance.

usandokan wrote:
> Hello,
>
> I'm having trouble importing a LDIF file into an IPlanet Directory
> Server 4.16 (IPDS) on Solaris 8.
> The LDIF is large, it's size is 2.4 GB, so I suspect that the problem
> is the inability of IPDS to open files larger than 2 GB.
>
> I want to confirm that there is a limitation of the software to open
> large files, before to get into the effort to split the file in
> smaller pieces.
>
> If so, is there some way to avoid the limitation?...
>
> Any help will be greatly appreciated.
>
> Thank you in advance.

I think you'll need to split the LDIF anyway, because IPDS makes
log-files for every LDIFfile/LDIFchunk imported, and these logfiles grow
big quite fast when doing big LDIF imports. We had the problem of them
filling up a filesystem when importing only 7000 entries. I don't have
the details about the filesystem size, and if the LDIF imported was
split into chunks, but can get them for you it you're interested. I
remember than we eventuelly split the file into several files with about
1000 records each and the logfiles stayed under 1/20th of the size of
the filesystem, so they seem to grow more than proportional to the
number or entries imported.

HTH, Erik.
--
---------------------------------------------------------------------------
Erik C.J. Laan elaan at dds.nl
Please reply below the message, please cut unrelevant pieces from a reply.
---------------------------------------------------------------------------

Hello Erik,

Thank you for your comments, I agree with you that I can't get out of
spliting the file.

I have the doubt, when exporting I have some option with the db2ldif
utility to make smaller chunks of data?

Because I've been thinking about one program to read the ldif and
split the file with N number of entries, but maybe there is a smarter
way to achieve this .

Also, I'll take care about the log files, to keep space in the file
system.

Thank you.

> I think you'll need to split the LDIF anyway, because IPDS makes
> log-files for every LDIFfile/LDIFchunk imported, and these logfiles grow
> big quite fast when doing big LDIF imports. We had the problem of them
> filling up a filesystem when importing only 7000 entries. I don't have
> the details about the filesystem size, and if the LDIF imported was
> split into chunks, but can get them for you it you're interested. I
> remember than we eventuelly split the file into several files with about
> 1000 records each and the logfiles stayed under 1/20th of the size of
> the filesystem, so they seem to grow more than proportional to the
> number or entries imported.
>
> HTH, Erik.

usandokan wrote:
> Hello Erik,
>
> Thank you for your comments, I agree with you that I can't get out of
> spliting the file.
>
> I have the doubt, when exporting I have some option with the db2ldif
> utility to make smaller chunks of data?
>
> Because I've been thinking about one program to read the ldif and
> split the file with N number of entries, but maybe there is a smarter
> way to achieve this .
>
> Also, I'll take care about the log files, to keep space in the file
> system.
>
>>I think you'll need to split the LDIF anyway, because IPDS makes
>>log-files for every LDIFfile/LDIFchunk imported, and these logfiles grow
>>big quite fast when doing big LDIF imports. We had the problem of them
>>filling up a filesystem when importing only 7000 entries. I don't have
>>the details about the filesystem size, and if the LDIF imported was
>>split into chunks, but can get them for you it you're interested. I
>>remember than we eventuelly split the file into several files with about
>>1000 records each and the logfiles stayed under 1/20th of the size of
>>the filesystem, so they seem to grow more than proportional to the
>>number or entries imported.

I think db2ldif talks about LDIF-chunks. That are set of
ldif-(change)-record" followed of by a line with only "-" and an empty
line. I think IPDS starts to actually do the database update only after
the chunk has been read and parsed. I think log-files get written for
every chunk. So making smaller chunks will make sure smaller log-files
are written. So if the above is correct this will not help you if the
filesize is over 2GB. Making some LDIF chunks will probably even make
the actual file (slightly) larger, as more "chunk-delimiters" (- & empty
line) are added.

I don't have enough experience with db2ldif to know from my head if it
will make separate ldif-files if you ask it to split the output-file
into multiple chunks, and I don't have an installation at hand to check.
Maybe it will, and using this options also solves your largefile issue.

HTH, Erik
--
---------------------------------------------------------------------------
Erik C.J. Laan elaan at dds.nl
Please reply below the message, please cut unrelevant pieces from a reply.
---------------------------------------------------------------------------