zorba-coders team mailing list archive

[Rodolfo Ochoa <rodolfo.ochoa@xxxxxxxxx>]

Rodolfo Ochoa has proposed merging lp:~zorba-coders/zorba/bug935639 into lp:zorba.

Requested reviews:
  Cezar Andrei (cezar-andrei)
  Matthias Brantner (matthias-brantner)
Related bugs:
  Bug #935639 in Zorba: "Closing without releasing objects"
  https://bugs.launchpad.net/zorba/+bug/935639

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/bug935639/+merge/94900

The problem can't be solved by technical means, the developer have to take care of resource usage, as in C++ the user must release all objects before shutdown the store, the same thing applies for XQJ.
So. the best workaround this problem is to show a more friendly message around the Strings Pool and create documentation that explain this issue. At the end, this issue affects every API usage from our clients.
Please check my changes and help me with English usage in the new document I created.
-- 
https://code.launchpad.net/~zorba-coders/zorba/bug935639/+merge/94900
Your team Zorba Coders is subscribed to branch lp:zorba.

=== modified file 'doc/zorba/indexpage.dox.in'
--- doc/zorba/indexpage.dox.in	2012-01-03 18:03:54 +0000
+++ doc/zorba/indexpage.dox.in	2012-02-28 02:26:20 +0000
@@ -29,6 +29,8 @@
     - \ref ft_thesaurus
   - \ref errors_warnings
   - \ref impl_dep_features
+  - \ref memory_management
+  - \ref memory_leaks
 
 \section xqlib XQuery Modules
  - <a href="../xqdoc/xhtml/index.html">XQuery Module Library</a>

=== added file 'doc/zorba/memory_leaks.dox'
--- doc/zorba/memory_leaks.dox	1970-01-01 00:00:00 +0000
+++ doc/zorba/memory_leaks.dox	2012-02-28 02:26:20 +0000
@@ -0,0 +1,146 @@
+/** \page memory_leaks Memory Leaks
+
+\section memory_leaks_intro Introduction
+
+The Zorba XQuery processor manage automatically its own memory, but still, there are cases where you may receive the following message:
+
+\code
+Zorba did not close properly, objects may still in memory while shutdown the store. For help avoiding this message please refer to http://www.zorba-xquery.com/memory_leaks.
+ID: 1   Referenced URI:  [..uri...]
+...
+
+Zorba Internal Fatal Error in ....
+[n] referenced URIs remain in the string pool.
+\endcode
+
+
+This message means the store is being shutdown, but there are still URIs strings in the pool because 
+there are still references (smart pointers) to those strings.  
+The references may be somewhere in the zorba engine (which usually means there is a memory leak) or in the application. 
+For the application, the rule is that before shutting down the store, 
+all variables storing references to zorba objects must have gone out of scope, or explicitly set to NULL.
+
+So, strictly speaking, your program is leaking resources.
+
+There are several cases using Zorba's APIs in C++ or other languages where you can produce this error besides everything looks correct in your code, here is an example:
+
+
+\section memory_leaks_example_c Example in C++
+Here is a code example that shows how a StaticContext and a Query are pointing to the URI http://www.foo.com, 
+and they are still in memory when the store is being shutdown.
+
+\code
+int main(int argc, char* argv[])
+{
+  void* lStore = zorba::StoreManager::getStore();
+  Zorba* lZorba = Zorba::getInstance(lStore);
+  
+  StaticContext_t lContext = lZorba->createStaticContext();
+  lContext->addNamespace("foo", "http://www.foo.com";);
+  XQuery_t lQuery = lZorba->compileQuery("<foo:e/>", lContext);
+  lQuery->execute();
+  
+  lZorba->shutdown();
+  zorba::StoreManager::shutdownStore(lStore);
+  return 0;
+}
+\endcode
+
+So, for this scenario, the ideal way to solve the problem is through scopes, 
+where you implement certain part of code in methods and release them when the objects loose their reference. 
+Other way to do it, is to explicitely set the object to null, the autopointer automatically will free the object.
+
+Solution example:
+\code
+int main(int argc, char* argv[])
+{
+  void* lStore = zorba::StoreManager::getStore();
+  Zorba* lZorba = Zorba::getInstance(lStore);
+  
+  {  // Use a scope
+
+  StaticContext_t lContext = lZorba->createStaticContext();
+    lContext->addNamespace("foo", "http://www.foo.com";);
+    XQuery_t lQuery = lZorba->compileQuery("<foo:e/>", lContext);
+    lQuery->execute();
+    lContext = NULL;  // or explicitely free the resource
+    lQuery = NULL;
+  }
+  lZorba->shutdown();
+  zorba::StoreManager::shutdownStore(lStore);
+  return 0;
+}
+\endcode
+
+
+\section memory_leaks_example_java Example in Java
+This particular error is specially notorious when a memory managed language 
+shows the error because you expect the language frees all memory, here is an example:
+
+\code
+  public static void main ( String argv[] )
+  {
+    InMemoryStore store = InMemoryStore.getInstance();
+    Zorba zorba = Zorba.getInstance ( store );
+
+    StaticContext context = zorba.createStaticContext();
+    context.addNamespace("foo", "http://www.foo.com";);
+    XQuery query = zorba.compileQuery("<foo:e/>", context);
+    String result = query.execute();
+    
+    zorba.shutdown();
+    InMemoryStore.shutdown ( store );
+    
+  }    
+\endcode
+
+In this example, and for the rest of the languages because Zorba is created in C++, 
+we have created in the Zorba API the method \c destroy() \e that will set free the object
+that could be pointing to any resource from the store, this method is in every object that need to be released.
+
+*Java Note: Because Java is a garbage collected language you cannot predict when (or even if) an object will be destroyed. 
+Hence there is no direct equivalent of a destructor.
+There is an inherited method called finalize, but this is called entirely at the discretion of the garbage collector.
+So, destroy() is the best practice for any language including Java.
+
+Solution example:
+\code
+  public static void main ( String argv[] )
+  {
+    InMemoryStore store = InMemoryStore.getInstance();
+    Zorba zorba = Zorba.getInstance ( store );
+
+    StaticContext context = zorba.createStaticContext();
+    context.addNamespace("foo", "http://www.foo.com";);
+    XQuery query = zorba.compileQuery("<foo:e/>", context);
+    String result = query.execute();
+    query.destroy();   // Release memory for this XQuery
+    context.destroy(); // Release memory for this StaticContext
+    
+    zorba.shutdown();
+    InMemoryStore.shutdown ( store );
+    
+  }    
+\endcode
+
+
+\section memory_leaks_example_xqj Example in XQJ
+
+XQJ standard provides specific \c close() \e methods for this specific purpose:
+\code
+      XQDataSource xqdatasource = new XQDataSource();
+      XQConnection xqconnection = xqdatasource.getConnection();
+      XQStaticContext staticContext = xqconnection.getStaticContext();
+      staticContext.declareNamespace("foo", "http://www.foo.com";);
+      xqconnection.setStaticContext(staticContext);
+      XQExpression xqexpression = xqconnection.createExpression();
+      XQSequence xqsequence = xqexpression.executeQuery("<foo:e/>");
+      // code to show the output
+      xqconnection.close();  // Closing connection and freeing all related resource
+\code
+
+\endcode
+
+
+*/
+/* vim:set et sw=2 ts=2: */

=== modified file 'src/store/naive/string_pool.cpp'
--- src/store/naive/string_pool.cpp	2012-02-15 10:25:02 +0000
+++ src/store/naive/string_pool.cpp	2012-02-28 02:26:20 +0000
@@ -26,20 +26,29 @@
 ********************************************************************************/
 StringPool::~StringPool() 
 {
+  bool stringsInPool = false;
   ulong count = 0;
   ulong n = (ulong)theHashTab.size();
-  for (ulong i = 0; i < n; i++)
+  for (ulong i = 0; i < n, !stringsInPool; i++)
   {
     if (theHashTab[i].theItem.is_shared())
     {
-      std::cerr << "i = " << i << " String " << theHashTab[i].theItem
-                << " is still in the pool" << std::endl;
+      stringsInPool = true;
+    }
+  }
+  if (stringsInPool) {
+    std::cerr << "Zorba did not close properly, objects may still in memory. For help avoiding this message please refer to http://www.zorba-xquery.com/html/documentation in section General Architecture -> Memory Leaks." << std::endl;
+    for (ulong i = 0; i < n; i++)
+    {
+      if (theHashTab[i].theItem.is_shared())
+      {
+      std::cerr << "ID: " << i << "   Referenced URI: " << theHashTab[i].theItem << std::endl;
       //delete theHashTab[i].theString.getp();
-      count++;
+      count ++;
+      }
     }
+    ZORBA_FATAL(stringsInPool==false, count << " referenced URI(s) remain in the string pool.");
   }
-
-  ZORBA_FATAL(count == 0, count << " strings remain in the string pool");
 }