zorba-coders team mailing list archive

[Matthias Brantner <1016053@xxxxxxxxxxxxxxxxxx>]

*** This bug is a duplicate of bug 898168 ***
    https://bugs.launchpad.net/bugs/898168

To circumvent the problem, you could try doing it as follows:

for $i at $y in parse-xml:parse(
      fetch:content("wiki.xml"),
      <opt:options>
        <opt:parse-external-parsed-entity opt:skip-root-nodes="1" />
      </opt:options>)
return {
  if ($i mod 1000 eq 0)
    cdml:apply-insert-nodes( $def:articles, $w[fn:last()] );
  else
    ();
}

** This bug has been marked a duplicate of bug 898168
   Windowing doesn't stream

** Changed in: zorba
     Assignee: Matthias Brantner (matthias-brantner) => Markos Zaharioudakis (markos-za)

-- 
You received this bug notification because you are a member of Zorba
Coders, which is the registrant for Zorba.
https://bugs.launchpad.net/bugs/1016053

Title:
  Memory overload when streaming large file

Status in Zorba - The XQuery Processor:
  New

Bug description:
  We tried to put every 1000th wikipedia article into a Zorba
  collection. Unfortunately, the computer's main memory quickly fills up
  and swapping starts at ~93% memory consumption for the zorba process.

  Code is attached. The wiki.xml file (36GB) is a recent Wikipedia Dump
  and can be obtained at http://download.wikimedia.org/enwiki/latest
  /enwiki-latest-pages-articles.xml.bz2 (~7.8GB when compressed). There
  are currently around 4,000,000 articles in the English Wikipedia.

To manage notifications about this bug go to:
https://bugs.launchpad.net/zorba/+bug/1016053/+subscriptions