pbxt-discuss team mailing list archive

[Michael Han <mh6@xxxxxxxxxxxx>]

On 7 Jul 2009, at 09:47, Paul McCullagh wrote:
> Hi Michael,
>
> Do you have any details on the issues? Did the remap just fail?
>
> As far as I can tell, a remap results in all memory blocks being  
> used for the file becoming invalid.
>
> If this is the case, then the operation is way too expensive with  
> the file grows over 1GB.
>
> On Jul 6, 2009, at 1:30 PM, Michael Han wrote:
>
> > Hi Paul,
> >
> > I actually had some issues with the remapping in cases where the  
> > files grow larger than the main memory of the servers (don't ask).
> >
> > But I can see the advantage of using mmap.
> >
> > Michael

I was importing a dump of some tables with a couple of million rows  
(5.9 million to be exact, got rid of the indices before loading). I  
could basically watch the file grow to 2.5 gigs (which is the main  
memory on the test server) and stall.

In the logs I got error messages with PBXT (1.0.0.08) trying to  
reallocate memory, failing and ending up in an endless loop.

Feeling-wise it was because it tried to map the file, couldn't map it  
into half main / half swap, and failed.

I got rid of the logs, but I  can try it again, as i still have the  
dumps and binaries flying around.

Michael


--
The Wellcome Trust Sanger Institute is operated by Genome Research 
Limited, a charity registered in England with number 1021457 and a 
company registered in England with number 2742969, whose registered 
office is 215 Euston Road, London, NW1 2BE.