zorba-coders team mailing list archive
-
zorba-coders team
-
Mailing list archive
-
Message #09854
[Merge] lp:~zorba-coders/zorba/new-groupby into lp:~zorba-coders/zorba/post-25
Carlos Manuel Lopez has proposed merging lp:~zorba-coders/zorba/new-groupby into lp:~zorba-coders/zorba/post-25.
Requested reviews:
Markos Zaharioudakis (markos-za)
For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/new-groupby/+merge/106070
Reworked group by clauses to work as the spec specifies atm.
--
The attached diff has been truncated due to its size.
https://code.launchpad.net/~zorba-coders/zorba/new-groupby/+merge/106070
Your team Zorba Coders is subscribed to branch lp:~zorba-coders/zorba/post-25.
=== modified file 'ChangeLog'
--- ChangeLog 2012-04-25 17:16:48 +0000
+++ ChangeLog 2012-05-16 22:51:21 +0000
@@ -1,47 +1,60 @@
Zorba - The XQuery Processor
+
+version 2.x
+
+ * Fixed bug #867170 (Variables declared in Prolog are Overwritten inside the query)
+
+
version 2.5
New Features:
- * fn:available-environment-variables
- * fn:environment-variables
- * fn:uri-collection
- * fn:unparsed-text
- * fn:unparsed-text-available
+ * XQuery 3.0 functions
+ - fn:available-environment-variables
+ - fn:environment-variables
+ - fn:uri-collection
+ - fn:unparsed-text
+ - fn:unparsed-text-available
+ - fn:has-children#0
+ - fn:nilled#0
+ - fn:path
* Extended API for Python, Java, PHP and Ruby.
* Add jvm classpath to zorbacmd and to Zorba API. Tracked by #931816
+ * Added full-text module.
* Added support for NO_ICU (to not use ICU for unicode processing)
* Added XQJ support.
-
-Optimization:
+ * Added CollectionManager and DocumentManager support for XQJ.
Bug Fixes/Other Changes:
+ * Fixed bugs #931501 and #866987 (improved error messages for fn:format-number(). Additionally, the function now throws the FODF1310 error instead of XTDE1310, as the 3.0 spec requires)
* Fixed bug 955170 (Catch clause with URILiteral-based wilcard NameTest)
+ * Fixed memory leak in case of index truncation
+ * Fixed bug #862971 (no error upon duplicate function declarations)
+ * Fixed bug in mergeUpdates() method
* Fixed bug 955135 (err:XQDY0044 not caught by try-catch expressions)
- * Fixed bug #986075 (encountering flwor expr with no clauses; due to common
- subexression being formed when inlining var in if-then-else expression)
+ * Fixed bug #986075 (encountering flwor expr with no clauses; due to common subexression being formed when inlining var in if-then-else expression)
* Fixed bug #967864 (var substitution did not update theFreeVars property)
- * Fixed buf #891650 (context size var not always declared within path expr)
+ * Fixed bug #891650 (context size var not always declared within path expr)
* Fixed bug #948879 (--uri-path doesn't work with fetch:content())
+ * Fixed bug in MarkNodeCopyProps rule (static cast to replace_expr without chaing the expr kind first)
* Fixed bug in window iterator (binding the end vars in the output tuple stream)
* Fixed bug #866547 (protect index-join rule from general flwor)
* Fixed bug #867253 (cdml:delete-nodes should only remove root nodes)
* Fixed bug #967428 (do not hoist index creation outside a try-catch)
* Fixed performance problem with the findNodeSources function of the no-copy rule
* Fixed bug #872234 (prevent a rewritting to take place in case of sequential expr)
+ * Fixed bug #966706 (key uniqueness of index not enforced during incremental refresh)
* Fixed bug #906494 (default compile with D_FILE_OFFSET_BITS=64)
* Fixed bug #988412 (date:current-dateTime daylight saving)
* Fixed bug #912586, #912593 and #912722 (assertion failures with lax validation)
* Fixed bug #921458 (file:read-text-lines() blocking)
- * Fixed bug #981405 (do not hoist expr containing try-catch variables out of the
- associated try-catch expression)
- * Fixed bug #947627 (throw XQST0099 if more than one declarations of context item
- type in same module)
+ * Fixed bug #981405 (do not hoist expr containing try-catch variables out of the associated try-catch expression)
+ * Fixed bug #947627 (throw XQST0099 if more than one declarations of context item type in same module)
* Fixed bug #980526 (no-copy rule bug due to global var being set in "distant" udf)
* Fixed bug #949910 (has-children may be invoked on all nodes). Internally, zorba::store::Item::getChildren() now returns NULL on node classes without offspring (instead of raising an error).
- * Fixed Bug #933490: Error ItemFactoryImpl::createBase64Binary with istream
+ * Fixed Bug #933490 (Error ItemFactoryImpl::createBase64Binary with istream)
* Fixed bug #867112 (Diagnostic Handler was not working on external APIs)
-
+ * Fixed bug #857842 (Assertion failed with simple content element with comments)
version 2.2
@@ -89,6 +102,8 @@
* Fixed bug 867509 (Can not handle largest xs:unsignedLong values)
* Fixed bug 924063 (sentence is incorrectly incremented when token characters end without sentence terminator)
* Fixed bug 909126 (bug in cloning of var_expr)
+ * Fixed bug 928631 (external builtin function were not executed in the module they
+ were declared)
* Fixed bug in destruction of exit_catcher_expr
* Fixed bug #867024 (error messages)
* Fixed bug #957580 (stream read failure in StringToCodepointsIteartor)
=== modified file 'bin/test/mymod.xq'
--- bin/test/mymod.xq 2012-04-24 12:39:38 +0000
+++ bin/test/mymod.xq 2012-05-16 22:51:21 +0000
@@ -14,7 +14,7 @@
: limitations under the License.
:)
-module namespace foo = "http://www.28msec.com/foo";;
+module namespace foo = "http://www.zorba-xquery.com/foo";;
declare function foo:test() {
if (fn:true())
=== modified file 'cmake_modules/FindICU.cmake'
--- cmake_modules/FindICU.cmake 2012-04-24 14:35:54 +0000
+++ cmake_modules/FindICU.cmake 2012-05-16 22:51:21 +0000
@@ -28,6 +28,8 @@
# (note: in addition to ICU_LIBRARIES)
# ICU_DATA_LIBRARIES - Libraries to link against for ICU data
#
+# ICU_VERSION - ICU's version number.
+#
# Look for the header file.
find_path(
=== modified file 'doc/zorba/build.dox'
--- doc/zorba/build.dox 2012-04-24 12:39:38 +0000
+++ doc/zorba/build.dox 2012-05-16 22:51:21 +0000
@@ -1,254 +1,18 @@
/** \page build Zorba Build Instructions
-\section requirements Requirements
-
-\subsection buildsystem Build System
-
-In order to build Zorba, you need the Cross-Platform Make CMake 2.6 or later
-(except CMake 2.6.3 that has a serious bug preventing Zorba from building).
-You can download CMake from http://www.cmake.org/.
-
-\subsection compilers Compilers
-Zorba is tested with the following compilers:
-- GNU Compiler: GCC 3.4.x (32-bit & 64-bit) and GCC 4.x.x
-- Microsoft Compiler: MS VC++ 2008, and MS VC++ 2010 (including Express)
-
-\subsection required_libs Required Libraries
-In order to build Zorba, you need the following libraries and development headers:
-
-\subsubsection required_core_zorba Required for Core Zorba
-The following packages are mandatory to build the core part of Zorba.
-
-<ul>
- <li>Libxml2 2.2.16 or later (http://xmlsoft.org/)</li>
- <li>Iconv 1.12 (used by Libxml2; http://www.gnu.org/software/libiconv/)</li>
- <li>ICU4C 3.6 or later (http://www.icu-project.org/)</li>
-</ul>
-
-\subsubsection optional_core_zorba Optional for Core Zorba
-The following packages are only required to build Zorba with some features (e.g. XML Schema or HTTP support).
-
-<ul>
- <li>Xerces-C 2.8.0, 3.0.0 or later (http://xerces.apache.org/xerces-c/). This package is only required if built
- with XML schema support. This feature is enabled by default. In order to disable XML schema support, you need
- to provide the CMake configuration variable \c ZORBA_NO_XMLSCHEMA=ON.
- </li>
- <li>CURL 7.12 or later (http://curl.haxx.se/). This package is only required if
- <a href="../../zorba/xqdoc/xhtml/www.zorba-xquery.com_modules_http-client.html">Zorba's HTTP module</a>
- should be available.
- Zorba will automatically be built with this module if the curl packages are installed. In order to suppress
- HTTP support even though curl is installed, please set the CMake configuration variable \c ZORBA_SUPPRESS_CURL=ON.
- Please note that other modules depend on this module (i.e. import it). Hence, those depending modules
- will not work if the http module is not available. Please see our
- <a href="../../zorba/xqdoc/xhtml/images/modules.svg">module dependency graph</a> for the
- module inter-dependencies.
- </li>
- <li>LibXslt version 1.1.24 or later (http://xmlsoft.org/XSLT/). The LibXslt package is required for XQueryX support
- (XQuery 1.0, XQuery 3.0, XQuery Update, and XQuery Full Text). This feature is not enabled by default.
- It can be enabled if Zorba is compiled using the the CMake configuration variable \c ZORBA_XQUERYX=ON.
- </li>
- <li>Flex 2.5.33 or later (http://flex.sourceforge.net/) and Bison 2.4 or later (http://www.gnu.org/software/bison/)
- are required to generate the XQuery scanner and XQuery parser. These packages are optional and only required
- if you want to make modifications to the scanner or parser, respectively.
- </li>
-</ul>
-
-\subsubsection required_language_bindings Required for Language Bindings
-In order to build the various language bindings that come with Zorba, you need
-at least Swig 1.3.40 or later (http://www.swig.org/download.html).
-All bindings are generated using Swig.
-
-For a particular language binding, you need the corresponding language tools
-and development headers.
-
-- Java (http://java.sun.com/javase/downloads/index.jsp)
-- Ruby (http://www.ruby-lang.org/en/downloads/)
-- PHP (http://www.php.net/downloads.php)
-- Python (http://www.python.org/download/)
-
-
-\section buildingzorba Building Zorba
--# Install the Zorba source distribution (see \ref installation).
- The directory in which the Zorba sources are installed
- is referred to as \c [ZORBA] in the following.
--# If desired, also install or download the source for any non-core modules
- that you would like to build along with Zorba, and ensure that you have
- any libraries required by those modules available. See \ref build_noncore
- for more details.
--# Zorba requires an
- <a href="http://www.cmake.org/Wiki/CMake_FAQ#What_is_an_.22out-of-source.22_build.3F"; target="_blank">out-of-source build</a>.
- We suggest to create the directory \c [ZORBA]/build
- and refer to this directory as \c [ZORBABUILD] in the following.
--# Change the working directory into the \c [ZORBABUILD] directory
- and execute cmake as follows:
- <tt>cmake [ZORBA]</tt>.
- In case the \c [ZORBABUILD]
- is located directly within the \c [ZORBA] directory,
- you can just type <tt>cmake ..</tt> .
- This command should configure Zorba and prepare for the build.
- CMake will tell you if your installation is missing
- some of the required libraries or development headers.
--# If CMake executed successfully,
- you should be able to build the project. For Makefile-based builds,
- just type \c make (or \c make -j2 to do a parallel build on a multi-core
- machine). For IDE-based builds, open the project created in the previous
- step and build the \c ALL target.
- The build will take some time.
- If it finishes successfully,
- you're ready to install and run Zorba
- (see \ref installation).
-
-\subsection buildoptions Build Options
-
-- CMake is a meta build system,
- meaning that it is able to generate native makefiles
- (e.g. GNU Make or NMake)
- and workspaces
- (e.g. KDevelop or Visual Studio Projects).
- For example,
- you can create a KDevelop Project by executing the following command
- in the \c [ZORBA] directory:
- <tt>cmake -G KDevelop3 [ZORBABUILD]</tt>.
-- CMake supports multiple build configurations
- (e.g. Debug, Release, or MinSizeRel).
- By default,
- Zorba is built in the Release configuration.
- To change the build mode (to Debug, Release, RelWithDebInfo or MinSizeRel),
- you can pass an additional parameter to CMake, e.g.,
- <tt>cmake -D CMAKE_BUILD_TYPE=Debug [ZORBA]</tt>.
-- Zorba has other build options as well
- (see \ref build_options).
- You can tweak the performance and library footprint
- by enabling or disabling various features from Zorba.
-
-\section mac Notes for Mac OS X Users
-
-The easiest way to install the required packages (like CMake or Xerces-C) is
-to use Macports (http://macports.org/).
-
-Once all the required packages are installed you can execute CMake.
-By default,
-CMake on Mac OS X uses the Makefile generator.
-Alternatively,
-you can specify different generators (e.g., Xcode)
-by starting CMake using the -G option (e.g., -G Xcode).
-
-\subsection swig Swig Version
-Due to a bug in older swig version,
-Zorba with SWIG support requires SWIG
-version 1.3.40 or later.
-
-To do this,
-the swig package from macports should be installed
-and point cmake to use the new version:
-
-\code
-sudo port install swig
-cmake -DCMAKE_PREFIX_PATH=/opt/local ..
-\endcode
-
-\section windows Notes for Windows Users
-- In order to compile Zorba on Windows,
- you need a working Visual Studio (Express) installation (2008, 2010).
-- If you want to compile Zorba using Visual Studio 2008,
- you will need to install the
- <a href="http://www.microsoft.com/downloads/en/details.aspx?FamilyId=D466226B-8DAB-445F-A7B4-448B326C48E7"; target="_blank">Visual C++ 2008 Feature Pack Release</a>.
-- Make sure that the libraries required for Zorba
- are built with the same version of Visual Studio that you use.
-- The Windows version of some required packages
- are distributed by other web sites
- (all of them are pointed to from the main package web sites
- indicated in the \ref required_libs section):
- - Libxml2 and Iconv: http://www.zlatkovic.com/libxml.en.html
- - CURL: http://curl.haxx.se/download.html
- - Libxslt: http://www.zlatkovic.com/pub/libxml/
-
-- Zorba has in place an automatic DLL detection mechanism.
- This will try to automatically gather all the DLLs
- from the third party librbaries
- and install them with Zorba when you do "name install"
- or you build the Visual Studio INSTALL project.
- Zorba will search for the third party libraries in order in:
- - all the directories pointed by the \c ZORBA_THIRD_PARTY_REQUIREMENTS
- cmake variable
- - e.g. if you have Xerces in "C:\tools" and IConv in "D:\tools",
- you must build with
- \code
- -D ZORBA_THIRD_PARTY_REQUIREMENTS=C:\tools;D:\tool
- \endcode
- and Zorba will find all it needs from Xerces and Iconv.
- (The directory names must contain "xerces" and "iconv" respectively, case insensitive)
- - in the program files directory on your system
- - e.g. if you have cURL installed in "C:\Program Files\*cURL*",
- Zorba will find all it needs from cURL.
- (The directory must contain "curl" case insensitive.)
- - all the paths in the \c PATH variable
- - e.g. if you have "C:\tools\*ICU*\bin" in your path,
- Zorba will find all it needs from ICU.
- (The directory must only contain "icu".)
-- Note: you can also use CMake's graphical interface.
- This can usually be found in the \e Start menu.
- This eases the setup and makes it more intuitive.
-
-\subsection nmake Building Zorba using a NMake Project
--# Start a Visual Studio Command Line.
--# Change the current working directory to the \c [ZORBA] directory.
--# It is not allowed to have
- <a href="http://www.cmake.org/Wiki/CMake_FAQ#What_is_an_.22out-of-source.22_build.3F"; target="_blank">in-source builds</a>.
- So you must create a build directory (e.g., <tt>[ZORBA]/build</tt>).
- We refer to this directory as \c [ZORBABUILD] in the following steps.
--# Change the current working directory to the \c [ZORBABUILD] directory.
--# Execute CMake as follows <tt>cmake -G "NMake Makefiles" [ZORBA]</tt>.
- CMake will try to find the necessary packages
- in the program files directory
- on your system and in the \c PATH environment variable. If one of the required dependencies were not found, CMake will stop with an error. See the next step only if this step did not find the required third party dependencies on your system.
--# If the command above fails or if you see that some optional packages were not found, you can help CMake a little by providing the lists of path where it should look for the libraries you need. In this case the CMake command should look like:
-\code
-"cmake" -G "NMake Makefiles" -D ZORBA_THIRD_PARTY_REQUIREMENTS=path\to\dir1;path\to\dir2;...
-\endcode
-
-But you can also manually set the paths to individual libraries and include directory by directly providing the variables set by each find package module (in the \c cmake_modules directory).
-
-In order to add XQueryX support you have to add ZORBA_XQUERYX=ON and add the paths to libxslt
-\code
--D ZORBA_XQUERYX=ON
--D LIBXSLT_INCLUDE_DIR="path_to_libxslt\include"
--D LIBXSLT_LIBRARIES="path_to_libxslt\lib\libxslt.lib"
-\endcode
-
-Now you are ready to call \c nmake from the \c [ZORBABUILD] directory.
-
-\subsection visualstudio Building Visual Studio Project
--# Start Visual Studio Command Line
--# Change the current working directory into the \c [ZORBA] directory.
--# Execute CMake as described above
- but with the appropriate CMake Visual Studio generator
- (<tt>-G "Visual Studio ..."</tt> instead of <tt>-G "NMake Makefiles"</tt>.
--# Start Visual Studio,
- open the generated \c zorba.sln project file,
- and start compilation.
-
-\subsection https_support HTTPS support
-- In order to have SSL support in Zorba,
- you have to use the CURL library that has SSL support.
-- There is one more thing:
- in order to validate the server's certificate,
- CURL+SSL needs to know about the root Certificates of Authenticity (CA):
- a set of public keys freely available on the internet.
-- On Windows,
- Zorba expects this file to be called \c cacert.pem
- and searches for it in the current directory and system paths.
-- This \c cacert.pem file can be found on the curl page:
- http://curl.haxx.se/docs/caextract.html .
-- In order to disable the validation of the server certificate,
- the \c ZORBA_VERIFY_PEER_SSL_CERTIFICATE can be set to \c OFF
- in cmake
- (and actually this is the default value).
- Set it to \c ON to validate server certificate chain
- on every access to \c https:// .
-- To read more about CURL+SSL: http://curl.haxx.se/docs/sslcerts.html .
-- As an \c https test page,
- we used for example the rss feed
- fn:doc('https://www.npr.org/rss/rss.php?id=1001') .
+The following pages describe how to build Zorba from source.
+
+- \ref build_prerequisites
+- \ref build_prepare
+- \ref configure_zorba
+- \ref build_and_install
+
+In addition, some platform-specific notes regarding building Zorba are
+available on the following pages. Please refer to them for
+clarifications and hints regarding the build process on your platform.
+
+- \ref build_ubuntu
+- \ref build_windows
+- \ref build_macos
*/
=== added file 'doc/zorba/build_and_install.dox'
--- doc/zorba/build_and_install.dox 1970-01-01 00:00:00 +0000
+++ doc/zorba/build_and_install.dox 2012-05-16 22:51:21 +0000
@@ -0,0 +1,59 @@
+/** \page build_and_install Building and Installing Zorba
+
+\section build_zorba Building Zorba
+
+Once you have successfully configured your Zorba build (as described
+in \ref configure_zorba), you should now be able to build the project.
+
+- For Makefile-based builds, just type \c make (or <tt>make j2</tt> to
+do a parallel build on a multi-core machine) from inside the \c
+{ZORBABUILD} directory.
+
+- For IDE-based builds, open the project created in the previous step
+ and build the \c ALL target.
+
+The build will take some time. If it finishes successfully, you're
+ready to install and run Zorba.
+
+\section install_zorba Installing Zorba
+
+- For Makefile-based builds, just type <tt>make install</tt>. There
+is also a <tt>make uninstall</tt> target.
+
+- For IDE-based builds, invoke the \c INSTALL project.
+
+In either case, Zorba will be installed into the location specified by
+the CMake configuration parameter \c CMAKE_INSTALL_PREFIX, as
+described in \ref configure_parameters.
+
+\section install_paths Setting the PATH
+
+In order for users on your system to use Zorba, the \c bin/ subdirectory
+of your installation directory must be on their path.
+
+\section testing_install Testing your Zorba Installation
+
+Once Zorba is installed and on your path, try it out! Type:
+
+\code
+zorba -q "1+1"
+\endcode
+
+You should get the following output:
+
+\code
+<?xml version="1.0" encoding="UTF-8"?>
+2
+\endcode
+
+If so - congratulations! You're done and ready to use Zorba.
+
+\section build_help Help!
+
+If you have any problems building or running Zorba, contact the \c
+zorba-users mailing list. Sign up and see other instructions at
+https://lists.sourceforge.net/lists/listinfo/zorba-users .
+
+
+*/
+
=== added file 'doc/zorba/build_configure.dox'
--- doc/zorba/build_configure.dox 1970-01-01 00:00:00 +0000
+++ doc/zorba/build_configure.dox 2012-05-16 22:51:21 +0000
@@ -0,0 +1,137 @@
+/** \page configure_zorba Configuring a Zorba Build Using CMake
+
+\section configure_overview Overview
+
+We will now configure the Zorba build by running CMake. We assume that
+the appropriate build preparations (described in \ref build_prepare)
+have been performed.
+
+\note
+The instructions in this section are based on executing commands at
+the command line. It is also possible configure Zorba using CMake's
+GUI configuration utility if you prefer. On Linux distributions, this
+is often a separate package named \"\c cmake-gui\". On Windows, the
+GUI is installed by default and can be started via the Start menu. On
+MacOS, it is available as the \"\c gui\" variant of the \c cmake
+package.
+
+\note
+The important configuration parameters - source and build directory;
+CMake generator; configuration flags - are the same whether you use
+the command-line or the GUI form of CMake. Only the method by which
+you specify them will change.
+
+The basic steps to configure Zorba are:
+
+-# Change the working directory into the \c {ZORBABUILD} directory:
+\code
+cd {ZORBABUILD}
+\endcode
+-# Execute cmake:
+\code cmake [ -G generator ] [ -D option=value ... ] {ZORBASRC}
+\endcode
+where \c {ZORBASRC} may be an absolute or a relative path to your
+Zorba source directory. For example, if you used the convention of
+creating the build directory as a subdirectory of the source
+directory, you can just type \code cmake .. \endcode
+along with any necessary \c -G or \c -D options.
+
+This command should configure Zorba and prepare for the build. CMake
+will tell you if your environment is missing some of the required
+libraries or development headers.
+
+The meaning of the \c -G and \c -D arguments are described below.
+
+\note
+If you need to re-configure the project later, CMake will remember
+your chosen generator and \c -D configuration parameters; it is not
+necessary to specify \c -G / \c -D every time.
+
+\section configure_generator CMake Generators (the -G argument)
+
+CMake is a meta build system. It is able to generate both native
+makefiles (e.g. GNU Make or NMake) and IDE workspaces (e.g. Visual
+Studio or KDevelop Projects), depending on which CMake \a generator is
+selected. By default it will generate a UNIX-style Makefile-based
+project.
+
+You specify the generator to use with the \c -G arguments to \c cmake.
+For example, on Windows, you can create a Visual Studio 10 solution by
+executing the following command in the \c {ZORBASRC} directory: \code
+cmake -G "Visual Studio 10" {ZORBASRC} \endcode
+
+The list of available Generators for your CMake installation
+can be seen by typing \" <tt>cmake --help</tt> \".
+
+\section configure_parameters Configuration Parameters (the -D arguments)
+
+In most circumstances it will be necessary to specify a few parameters to
+CMake, such as the various configuration parameters mentioned above in
+\ref third_party_libs. You do this with a series of \c -D arguments to
+CMake. For example, to build Zorba without ICU (Unicode) support:
+
+\code
+cmake -D ZORBA_NO_ICU=ON {ZORBASRC}
+\endcode
+
+A complete list of Zorba's configuration parameters can be found here:
+\ref configuration_parameters. CMake itself also has some
+configuration parameters that may be important for you; here are three
+of the most common:
+
+-# CMake will look for third-party libraries in a variety of standard
+ locations. If you have any installed in unusual locations, you will
+ need to provide it with additional paths to search. You do this by
+ specifying the parameter \c CMAKE_PREFIX_PATH. This parameter is a
+ semicolon-separated list of directories where CMake should look.
+ For instance, if you have Xerces-C installed in \c /opt, try the
+ following:
+\code
+cmake -D CMAKE_PREFIX_PATH=/opt {ZORBASRC}
+\endcode
+ \par On Linux and MacOS:
+ In command lines, the semicolon is
+ interpretted by the shell as a command separator. In order to pass a
+ semicolon-separated list of paths for this parameter, be sure to
+ enlose the list in single- or double-quotes.
+ \par On Windows:
+ Zorba uses a custom mechanism for specifying the paths to
+ third-party dependencies on Windows, which handles some additional
+ features such as collecting the runtime DLLs for installation. You
+ should use this mechanism instead of \c CMAKE_PREFIX_PATH on Windows. See
+ \ref windows_autodll for more information.
+
+-# After you build Zorba, you will likely want to install it into a
+ final location. You specify this location with the \c
+ CMAKE_INSTALL_PREFIX parameter. By default, this directory will be
+ \c {ZORBABUILD}/dist, which is not likely to be a useful location.
+ \par Note:
+ After installation, the directory specified here will contain
+ subdirectories such as \c bin/, \c lib/, and \c share/. On Unix and
+ MacOS installations, a common value for \c CMAKE_INSTALL_PREFIX is
+ \c /usr/local. On Windows, something like <tt>C:\\Program
+ Files\\Zorba</tt> is suggested.
+ \par Note for Makefile-based projects:
+ The Makefiles produced by CMake do support the common \c DESTDIR
+ variable. However, specifying this variable at compile time <em>will
+ not work</em> with Zorba, because certain installation paths are
+ hard-coded into the Zorba binaries. Be sure to only use \c
+ CMAKE_INSTALL_PREFIX.
+
+-# When generating a Makefile-based project, CMake supports multiple
+ build configurations. By default, Zorba is built in the Release
+ configuration, which enables compiler optimizations and does not
+ build debug information into the resulting product. To change the
+ build mode to Debug, you can specify the \c CMAKE_BUILD_TYPE
+ parameter as follows:
+ \code cmake -D CMAKE_BUILD_TYPE=Debug {ZORBASRC} \endcode
+ This is not necessary for IDE-based workspaces;
+ in that case, you may select the type of build from within the IDE.
+
+
+\section configure_next What's next?
+
+Once CMake runs without reporting any configuration errors, move on to
+\ref build_and_install.
+
+*/
=== added file 'doc/zorba/build_macos.dox'
--- doc/zorba/build_macos.dox 1970-01-01 00:00:00 +0000
+++ doc/zorba/build_macos.dox 2012-05-16 22:51:21 +0000
@@ -0,0 +1,33 @@
+/** \page build_macos Zorba Build Notes for Mac OS X Users
+
+\section macos_macports Using Macports
+
+The easiest way to install the required packages (like CMake or
+Xerces-C) is to use Macports (http://macports.org/). You can install
+various tools and libraries using e.g.
+\code
+sudo port install cmake
+\endcode
+
+Once all the required packages are installed you can execute CMake.
+By default,
+CMake on Mac OS X uses the Makefile generator.
+Alternatively,
+you can specify different generators (e.g., Xcode)
+by starting CMake using the \c -G option (e.g., <tt>-G Xcode</tt>).
+
+\section macos_swig Swig Version
+
+Due to a bug in older swig version, Zorba with SWIG support requires
+SWIG version 1.3.40 or later. SWIG is included in stock MacOS, but in
+at least some versions of MacOS it is an older version. Therefore, you
+must install a newer SWIG from Macports. This will be installed in
+<tt>/opt/local</tt>, so you then must point CMake at that directory to
+ensure it finds the newer version:
+
+\code
+sudo port install swig
+cmake -DCMAKE_PREFIX_PATH=/opt/local {ZORBASRC}
+\endcode
+
+*/
=== removed file 'doc/zorba/build_noncore.dox'
--- doc/zorba/build_noncore.dox 2012-04-24 12:39:38 +0000
+++ doc/zorba/build_noncore.dox 1970-01-01 00:00:00 +0000
@@ -1,218 +0,0 @@
-/** \page build_noncore Building Non-core Zorba Modules
-
-\section noncore_download Downloading source code for non-core modules
-
-Currently, the Zorba team does not provide source downloads for the
-non-core modules. However, there is a simple CMake script inside Zorba
-which will allow you to check out these modules from source control
-easily. Note that this requires having the Subversion utility available
-on your system.
-
-From your [ZORBA] directory, type
-\code
- cmake -Doutdir=../zorba_modules -Dmodname=NAME -P modules/DownloadModules.cmake
-\endcode
-
-to download the particular module package NAME (see \ref
-non_core_modules for a complete list of module packages and their
-contents), or
-
-\code
- cmake -Doutdir=../zorba_modules -Dallmodules=1 -P modules/DownloadModules.cmake
-\endcode
-
-to download all the modules. (On Windows, you will need to use backslashes instead of forward slashes in the above paths.)
-
-This will place the downloaded module source code in the directory
-\c ../zorba_modules, which is where the Zorba build will look for them by
-default. If you wish to download them to some other location, you may do
-so; in that case, when you configure the Zorba build with CMake, provide
-the \c -DZORBA_MODULES_DIR=/full/path/to/modules argument.
-
-\subsection noncore_dependencies Dependencies among Non-core Modules
-
-Please note that some of these modules depend on other modules. Most
-notably, the EXPath http-client module (in the \c http-client module
-package) depends on the \c tidy module, which is in the \c data-converters module package. So, if you download the \c http-client package, you must also download the \c data-converters package (and ensure that libtidy is installed, as mentioned below).
-
-Please see our <a
-href="../../zorba/xqdoc/xhtml/images/modules.svg">module-interdependency
-graph</a> for details.
-
-
-\section noncore_requirements Non-core Module Requirements
-For many of the non-core modules, you need additional libraries.
-All such modules are not built if the library (and development headers) they require are not available.
-Here is a list of the libraries required by the current non-core modules packages:
-
-- data-converters (JSon, CSV, HTML)
- - Tidy (http://tidy.sourceforge.net)
- - Jansson (http://www.digip.org/jansson/)
-- data-formatting (XSL-FO)
- - Java (http://java.sun.com/javase/downloads/index.jsp)
- - Apache FOP (http://xmlgraphics.apache.org/fop/download.html)
-- email
- - IMAP CClient (http://www.washington.edu/imap/)
-- geo
- - GEOS version 3.2.2 or later (http://trac.osgeo.org/geos/)
-- EXPath http-client
- - CURL 7.12 or later (http://curl.haxx.se/)
-- image
- - ImageMagick (http://www.imagemagick.org)
-- languages (XSLT)
- - LibXslt version 1.1.24 or later (http://xmlsoft.org/XSLT/)
-
-
-In many cases you can find binary packages for these dependencies, which
-will be the quickest and easiest way to meet the requirements. We have
-provided links or instructions for downloading binary packages for these
-libraries below, along with some instructions that we have found when
-building these libraries from source.
-
-
-\subsection buildcclient Building IMAP CClient
-
-Zorba provides email support using the CClient library part of the <a href="http://www.washington.edu/imap/"; target="_blank">UW IMAP toolkit</a>.
-
-\subsubsection cclientunix Unix/Linux/Mac OS X
-Notes:
-- There are some known issues with the CClient packages
- that come with different Linux distributions.
-- On x32 bit OpenSuse and also on x64 bit Ubuntu,
- we noticed that the CClient shared library is broken
- (undefined symbol: mm_dlog).
-- Due to that fact that Mark Crispin
- (the creator of CClient library)
- does not support CClient as a shared library but only as a static library
- (see <a href="http://www.washington.edu/imap/IMAP-FAQs/index.html#6.3"; target="_blank">FAQs shared library</a>),
- we strongly suggest you want to get the
- <a href="http://www.washington.edu/imap/"; target="_blank">UW IMAP toolkit</a>
- and compile it yourself.
-- On Linux 64-bit,
- you might discover a problem with the optional package cclient.
- E.g. on Ubuntu 64-bit, you might discover the following:
- \code
-Linking CXX shared library libsmtp.so
-/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/4.4.3/../../../../lib/libc-client.a(osdep.o): relocation R_X86_64_32 against
- `server_input_wait' can not be used when making a shared object; recompile with -fPIC
-/usr/lib/gcc/x86_64-linux-gnu/4.4.3/../../../../lib/libc-client.a: could not read symbols: Bad value
- \endcode
- To fix this,
- you need to compile CClient yourself.
- The commands to compile CClient correctly are:
- \code
-cd <some_directory>
-wget ftp://ftp.cac.washington.edu/imap/imap-2007e.tar.gz
-tar -xf imap-2007e.tar.gz
-cd imap-2007e
-make slx EXTRACFLAGS="-I/usr/include/openssl -fPIC"
- \endcode
-
-- Also please keep in mind that if SSL/TLS authentication is required
- by the SMTP server,
- then you first need to install
- <a href="http://www.openssl.org/"; target="_blank">OpenSSL</a>
- and configure CClient to use it.
-- Make sure the name of the library is prefixed by \c lib
- and suffixed with \c .a
- (for example \c libc-client.a on Linux/Unix
- or \c libc-client4.a on Mac OS X).
-
-Use the following extra CMake arguments when building Zorba:
-\code
--D "CCLIENT_INCLUDE=path_to_imap-2007e/c-client"
--D "CCLIENT_LIBRARY=path_to_imap-2007e/c-client/libc-client.a"
-\endcode
-
-Here are some quick suggestions to build CClient on Linux:
-- x32-bit Linux: <tt>make lnp</tt>
-- x64-bit Linux: <tt>make lnp EXTRACFLAGS="-I/usr/include/openssl -fPIC" EXTRAAUTHENTICATORS=gss</tt> in order to build with SSL and Kerberos support.<br />
-
-For more detailed build instructions,
-please see the
-<a href="http://www.washington.edu/imap/documentation"; target="_blank">UW IMAP toolkit "Server Documentation"</a>
-and the <a href="http://www.washington.edu/imap/IMAP-FAQs/index.html"; target="_blank">UW IMAP FAQs</a>.
-
-\subsubsection cclientwindows Windows
-
-You must build the
-<a href="http://www.washington.edu/imap/"; target="_blank">UW IMAP toolkit</a>.
-
-Use the following extra CMake arguments when building Zorba:
-\code
--D "CCLIENT_INCLUDE=path_to_imap-2007e\c-client"
--D "CCLIENT_LIBRARY=path_to_imap-2007e\c-client\release\cclient.lib"
-\endcode
-
-\subsection buildjansson Building Jansson
-
-Zorba provides Json manipulation support
-using the Jansson Library (http://www.digip.org/jansson/).
-
-\subsubsection janssonunix Unix/Linux/Mac OS X
-
-For Linux/Unix systems,
-download the Jansson library from http://www.digip.org/jansson/releases/.
-Then follow the instructions from
-http://www.digip.org/jansson/doc/2.0/gettingstarted.html#compiling-and-installing-jansson.
-
-The source use GNU Autotools
-(autoconf, automake, libtool),
-so compiling and installing is extremely simple:
-\code
-./configure
-make
-make check
-make install
-\endcode
-
-On Fedora starting with version 12, Jansson is also available as a package,
-so (as root) you can simply run:
-\code
-yum install -y jansson-devel
-\endcode
-
-Ubuntu packages are available from PPA of
-https://launchpad.net/~petri/+archive/ppa
-
-\subsubsection janssonwindows Windows
-
-For Windows please follow these steps:
-- download Jansson from https://github.com/akheron/jansson/zipball/master
-- rename src/jansson_config.h.win32 to src/jansson_config.h
-- replace \c snprintf with \c _snprintf in \c src/dump.c lines 188 and 202
- and \c load.c lines 96 and 108
-- replace \c json_strtoint with \c strtol in \c load.c line 481
-- add all headers and source files in src folder and create a static library
-
-There will be a new Jansson 2.0.1 release
-that will include the fixed described above.
-
-\subsection buildimagemagick Building ImageMagick
-
-Zorba provides image manipulation support
-using the ImageMagick Library (http://www.imagemagick.org).
-
-\subsubsection imagemagickunix Unix/Linux/Mac OS X
-
-For Linux/Unix systems,
-just use the package manager to install the
-magick++, magickwand, magickcore packages
-including the developer versions of these packages
-(that inlcude the header files).
-
-On OSX, use macports to install imagemagick.
-
-\subsubsection imagemagickwindows Windows
-
-For Windows,
-just download the ImageMagick windows binary (Q8-windows-dll) and install it.
-
-\subsection buildtidy Libtidy
-
-\subsubsection buildtidywindows Windows
-
-A binary installer for Tidy is available at: http://dev.int64.org/tidy.html
-
-
-*/
=== removed file 'doc/zorba/build_options.dox'
--- doc/zorba/build_options.dox 2011-10-31 21:31:40 +0000
+++ doc/zorba/build_options.dox 1970-01-01 00:00:00 +0000
@@ -1,35 +0,0 @@
-/** \page build_options Zorba Build Options
-
-\section ZORBA_WITH_BIG_INTEGER Big Integer Support (ZORBA_WITH_BIG_INTEGER)
-
-The Zorba XQuery processor
-by default
-has arbitrary precision
-for the \c xs:integer type.
-Compared to a C++ \c int,
-however,
-arbitrary precision integers
-are orders of magnitude slower.
-If arbitrary precision integers are not needed,
-they can be disabled
-by configuring Zorba
-with the
-\c ZORBA_WITH_BIG_INTEGER
-option set to \c OFF.
-
-When arbitrary precision integers are disabled,
-Zorba uses a C++ <code>long long</code>
-for \c xs:integer,
-the precision for which
-is dependent on your C++ implementation.
-Typically, however,
-the precision is at least 64 bits.
-However,
-in order to maintain the condition
-that the value space for \c xs:nonNegativeInteger
-is a subset of that of \c xs:integer,
-one less bit is allowed
-for \c xs:nonNegativeInteger.
-
-*/
-/* vim:set et sw=2 ts=2: */
=== added file 'doc/zorba/build_parameters.dox'
--- doc/zorba/build_parameters.dox 1970-01-01 00:00:00 +0000
+++ doc/zorba/build_parameters.dox 2012-05-16 22:51:21 +0000
@@ -0,0 +1,91 @@
+/** \page configuration_parameters Zorba Configuration Parameters
+
+This is a list of all published Zorba configuration parameters. You
+specify these parameters at CMake configuration time with the \c -D
+argument. See \ref configure_parameters for more information.
+
+\section configure_enable Enabling/Disabling Zorba Features
+
+\subsection ZORBA_WITH_BIG_INTEGER Big Integer Support
+
+The Zorba XQuery processor by default has arbitrary precision for the
+\c xs:integer type. Compared to a C++ \c int, however, arbitrary
+precision integers are orders of magnitude slower. If arbitrary
+precision integers are not needed, they can be disabled by configuring
+Zorba with the \c ZORBA_WITH_BIG_INTEGER option set to \c OFF,
+i.e. <tt>-D ZORBA_WITH_BIG_INTEGER=OFF</tt>
+
+When arbitrary precision integers are disabled, Zorba uses a C++
+<code>long long</code> for \c xs:integer, the precision for which is
+dependent on your C++ implementation. Typically, however, the
+precision is at least 64 bits. However, in order to maintain the
+condition that the value space for \c xs:nonNegativeInteger is a
+subset of that of \c xs:integer, one less bit is allowed for \c
+xs:nonNegativeInteger.
+
+\subsection ZORBA_NO_FULL_TEXT XQuery Full-Text Support
+
+Zorba by default supports the XQuery Full-Text specification
+(http://www.w3.org/TR/xpath-full-text-10/ ). If you wish to disable
+this support, configure with <tt>-D ZORBA_NO_FULL_TEXT=ON</tt>.
+
+\subsection ZORBA_XQUERX XQueryX Support
+
+Zorba by default does not build with support for XQueryX (which is an
+alternate representation of XQuery using pure XML). To enable this
+support, configure with <tt>-DZORBA_XQUERYX=ON</tt>. Note that
+building with XQueryX support requires LibXslt to be available.
+
+\subsection ZORBA_VERIFY_PEER_SSL_CERTIFICATE Enabling SSL Peer Verification
+
+By default, Zorba's HTTP module does not validate the SSL certificate
+sent to it by HTTPS servers. This is because not all systems have a
+complete or valid set of root certificates against which to validate
+said server certificates. If you wish to enable SSL peer validation,
+configure with <tt>-D ZORBA_VERIFY_PEER_SSL_CERTIFICATE=ON</tt>.
+
+\section configure_deps Disabling Zorba Library Requirements
+
+\subsection ZORBA_NO_ICU Disable ICU (Unicode) Support
+
+The ICU library provides Unicode functionality, and is required by
+default when building Zorba. To disable this requirement, configure
+with <tt>-D ZORBA_NO_ICU=ON</tt>.
+
+\subsection ZORBA_NO_XMLSCHEMA Disable Xerces (XML Schema) Support
+
+The Xerces-C library provides XML Schema functionality, and is
+required by default when building Zorba. To disable this requirement,
+configure with <tt>-D ZORBA_NO_XMLSCHEMA=ON</tt>.
+
+\subsection ZORBA_SUPPRESS_CURL Disable Curl (HTTP) Support
+
+The Curl library provides HTTP functionality. Zorba will automatically
+build with HTTP support (via the built-in \c http-client module) if it
+detects \c libcurl. To suppress this check (so you can build Zorba
+without HTTP support even if Curl is installed on your system),
+configure with <tt>-D ZORBA_SUPPRESS_CURL=ON</tt>.
+
+\section configure_buildparams Parameters for Compiling
+
+\subsection ZORBA_MODULES_DIR Specifying the Directory for Non-Core Modules
+
+By default, Zorba will automatically build any non-core module
+packages it finds in a directory named \c zorba_modules that is a
+sibling to the directory containing Zorba's own source (that is,
+\"<tt>../zorba_modules</tt>\" from the source directory). See \ref
+noncore_download for more information.
+
+If you wish to build non-core modules from a different location,
+configure with <tt>-D ZORBA_MODULES_DIR=/path/to/modules</tt>.
+
+\subsection ZORBA_THIRD_PARTY_REQUIREMENTS Specifying the Location of Third-Party Requirements on Windows
+
+On Windows, you should configure with <tt>-D
+ZORBA_THIRD_PARTY_REQUIREMENTS=/path1;/path2</tt> to indicate the
+path(s) to third-party libraries, instead of \c
+CMAKE_PREFIX_PATH. This is described in detail at \ref
+windows_autodll.
+
+*/
+/* vim:set et sw=2 ts=2: */
=== added file 'doc/zorba/build_prepare.dox'
--- doc/zorba/build_prepare.dox 1970-01-01 00:00:00 +0000
+++ doc/zorba/build_prepare.dox 2012-05-16 22:51:21 +0000
@@ -0,0 +1,104 @@
+/** \page build_prepare Preparing to Build Zorba
+
+The following outlines the common steps for building Zorba once the
+required and optional tools and libraries (as described in \ref
+build_prerequisites) have been installed.
+
+\section build_gettingsrc Getting the Zorba Source Code
+
+You may get the Zorba source code in one of two ways: downloading a
+released source distribution, or getting the code from our Launchpad
+code repository. The directory that the Zorba source code is placed
+into will be referred to as \c {ZORBASRC} in later instructions.
+
+\subsection download_source Getting a Source Code Release
+
+You may download the latest Zorba release source code as either a \c
+.zip or \c .tar.gz file from Launchpad:
+https://launchpad.net/zorba/+download .
+
+\subsection checkout_source Getting the Latest Code from Launchpad
+
+If you want the very latest and greatest Zorba source, you can check
+out our source code trunk from our Launchpad repository. This is
+actually a pretty "safe" option, compared to many projects, as Zorba
+attempts to enforce a "trunk is always releasable" policy through
+extensive regression testing on every trunk checkin.
+
+Launchpad uses the Bazaar distributed version control system from
+Canonical, the producers of Ubuntu Linux and Launchpad. So you will
+need to install Bazaar in order to check out our code
+repositories. Installers are available from Canonical's website,
+http://bazaar.canonical.com/ . Bazaar packages are also available for
+all Linux distributions (the package name may be "bazaar" or "bzr"),
+and in Macports for MacOS X.
+
+Once you have Bazaar installed, check out the Zorba trunk:
+
+\code
+bzr checkout --lightweight lp:zorba
+\endcode
+
+\section noncore_download Downloading source code for non-core modules
+
+Zorba offers a number of optional ("non-core") modules which provide
+additional functionality; see \ref non_core_modules for a list. When
+you build Zorba, you may also build any non-core modules at the same
+time, and these modules will then be available in your Zorba
+installation.
+
+Currently, the Zorba team does not provide source downloads for the
+non-core modules. However, there is a simple CMake script inside Zorba
+which will allow you to check out these modules from source control
+easily. Note that this requires having the Bazaar utility available
+on your system; see \ref checkout_source for more information.
+
+From your {ZORBASRC} directory, type
+\code
+ cmake -Doutdir=../zorba_modules -Dmodname=NAME -P modules/DownloadModules.cmake
+\endcode
+
+to download the particular module package NAME (see \ref
+non_core_modules for a complete list of module packages and their
+contents), or
+
+\code
+ cmake -Doutdir=../zorba_modules -Dallmodules=1 -P modules/DownloadModules.cmake
+\endcode
+
+to download all the modules. (On Windows, you will need to use backslashes instead of forward slashes in the above paths.)
+
+This will place the downloaded module source code in the directory
+<tt>../zorba_modules</tt>, which is where the Zorba build will look
+for them by default. If you wish to download them to some other
+location, you may do so; in that case, when you configure the Zorba
+build with CMake, provide the \c
+-DZORBA_MODULES_DIR=/full/path/to/modules argument.
+
+\note
+Please note that some of these modules depend on other modules. Most
+notably, the EXPath http-client module (in the \c http-client module
+package) depends on the \c html module, which is in the \c
+data-converters module package. So, if you download the \c http-client
+package, you must also download the \c data-converters package (and
+ensure that \c libtidy is installed; see \ref noncore_requirements).
+\note
+Please see our <a
+href="../../zorba/xqdoc/xhtml/images/modules.svg">module-interdependency
+graph</a> for details.
+
+\section create_build_dir Creating a Build Directory
+
+Zorba requires an <a
+href="http://www.cmake.org/Wiki/CMake_FAQ#What_is_an_.22out-of-source.22_build.3F";
+target="_blank">out-of-source build</a>, which means you need to
+create a separate (empty) build directory. A common convention is to
+create a directory named \"\c build\" in the Zorba source directory,
+but you may name it and place it anywhere you like. We refer to this
+directory as \c {ZORBABUILD} in the following steps.
+
+\section prepare_next What's next?
+
+Now, go on to \ref configure_zorba.
+
+*/
=== added file 'doc/zorba/build_prerequisites.dox'
--- doc/zorba/build_prerequisites.dox 1970-01-01 00:00:00 +0000
+++ doc/zorba/build_prerequisites.dox 2012-05-16 22:51:21 +0000
@@ -0,0 +1,138 @@
+/** \page build_prerequisites Prerequisites for Building Zorba
+
+The following tools and libraries are utilized by Zorba.
+
+\section cmake Configuration System
+
+In order to build Zorba, you need the cross-platform build system
+CMake, version 2.6 or later (except CMake 2.6.3 that has a serious bug
+preventing Zorba from building). CMake's homepage is
+http://www.cmake.org/, and this is the best source for the Windows
+CMake installer. It is also available packaged for most Linux
+distributions, as well as in Macports for MacOS builds.
+
+\section compilers Compilers
+
+Zorba is tested with the following compilers:
+- GNU Compiler: GCC 3.4.x (32-bit & 64-bit) and GCC 4.x.x
+- Microsoft Compiler: MS VC++ 2008, and MS VC++ 2010 (including Express)
+
+\section third_party_libs Library Dependencies
+
+Zorba uses a number of libraries from third parties. A few of these
+are required for all Zorba installations, but we have tried to keep
+this number as low as possible. Several others are required to enable
+certain Zorba features.
+
+\note The following descriptions sometimes reference "CMake
+configuration parameters". These are passed to CMake during the
+configuration step; see \ref configure_zorba.
+
+\subsection required_core_zorba Required for Core Zorba
+The following packages are mandatory to build the core part of Zorba.
+
+- Libxml2 2.2.16 or later (http://xmlsoft.org/)
+- Iconv 1.12 (used by Libxml2; http://www.gnu.org/software/libiconv/)
+
+\subsection recommended_core_zorba Recommended for Core Zorba
+
+The following packages are necessary to enable certain features in
+Zorba. These features are normally desired, so you will need to
+explicitly disable them in order to build without these libraries.
+
+- Xerces-C 2.8.0, 3.0.0 or later
+ (http://xerces.apache.org/xerces-c/). This package is required to
+ enable XML schema support. This feature is enabled by default. In
+ order to disable XML schema support, you need to provide the CMake
+ configuration parameter \c ZORBA_NO_XMLSCHEMA=ON.
+
+- ICU4C 3.6 or later (http://www.icu-project.org/). This package is
+ required to enable Unicode support as well as some other
+ internationalization features. This feature is enabled by
+ default. To build without ICU, you need to provide the CMake
+ configuration parameter \c ZORBA_NO_ICU=ON.
+
+\subsection optional_core_zorba Optional for Core Zorba
+
+The following packages are also necessary to enable certain features
+in Zorba. These features will automatically be disabled (or are not
+enabled by default) if the corresponding dependencies are not detected
+by CMake.
+
+- CURL (libcurl) 7.12 or later (http://curl.haxx.se/). This package is
+ required if <a
+ href="../../zorba/xqdoc/xhtml/www.zorba-xquery.com_modules_http-client.html">Zorba's built-in
+ HTTP module</a> should be available. Zorba will automatically be
+ built with this module if the curl packages are installed.
+ \note Please note that other modules depend on the http-client module
+ (i.e., import it). Hence, those dependent modules will not work if
+ the http module is not available. Please see our <a
+ href="../../zorba/xqdoc/xhtml/images/modules.svg">module
+ dependency graph</a> for the module inter-dependencies.
+ \note Also note that Zorba internally uses this module when resolving
+ http: URIs in queries, module/schema imports, and so on. If this
+ module is not installed, Zorba will be unable to download such
+ URIs from the web.
+ \note If you happen to have libcurl installed but would like to build
+ Zorba without HTTP support, you may set the CMake configuration
+ parameter \c ZORBA_SUPPRESS_CURL=ON.
+
+- LibXslt version 1.1.24 or later (http://xmlsoft.org/XSLT/). The
+ LibXslt package is required for XQueryX support. This feature is
+ not enabled by default. To enable XQueryX support, you need to
+ provide the CMake configuration parameter \c ZORBA_XQUERYX=ON.
+
+- Flex 2.5.33 or later (http://flex.sourceforge.net/) and
+ Bison 2.4 or later (http://www.gnu.org/software/bison/) are
+ used to generate the XQuery scanner and XQuery parser. The Zorba
+ source includes pre-generated versions of these, so Flex and Bison
+ are only required if you are developing Zorba itself and need to
+ modify the scanner or parser.
+
+\subsection required_language_bindings Required for Language Bindings
+In order to build the various language bindings that come with Zorba, you need
+at least Swig 1.3.40 or later (http://www.swig.org/download.html).
+All bindings are generated using Swig.
+
+For a particular language binding, you need the corresponding language
+tools and development headers. CMake will automatically enable the
+language bindings for any language that is detected.
+
+- Java (http://java.sun.com/javase/downloads/index.jsp)
+- Ruby (http://www.ruby-lang.org/en/downloads/)
+- PHP (http://www.php.net/downloads.php)
+- Python (http://www.python.org/download/)
+
+\subsection noncore_requirements Non-core Module Requirements
+For many of the non-core modules, you need additional libraries.
+All such modules are not built if the library (and development headers) they require are not available.
+Here is a list of the libraries required by the current non-core modules packages:
+
+- data-converters (CSV, HTML)
+ - Tidy (http://tidy.sourceforge.net)
+- data-formatting (XSL-FO)
+ - Java (http://java.sun.com/javase/downloads/index.jsp)
+ - Apache FOP (http://xmlgraphics.apache.org/fop/download.html)
+- email
+ - IMAP CClient (http://www.washington.edu/imap/)
+- geo
+ - GEOS version 3.2.2 or later (http://trac.osgeo.org/geos/)
+- EXPath http-client
+ - CURL 7.12 or later (http://curl.haxx.se/)
+- image
+ - ImageMagick (http://www.imagemagick.org)
+- languages (XSLT)
+ - LibXslt version 1.1.24 or later (http://xmlsoft.org/XSLT/)
+
+In many cases you can find binary packages for these dependencies, which
+will be the quickest and easiest way to meet the requirements. We have
+provided links or instructions for downloading binary packages for some
+libraries on the "Build Notes" pages for various platforms; see \ref build.
+
+
+\section prereq_next What's Next?
+
+Once you have obtained all the prerequisite tools and libraries for
+your installation, move on to \ref build_prepare.
+
+*/
=== modified file 'doc/zorba/build_ubuntu.dox'
--- doc/zorba/build_ubuntu.dox 2012-04-24 12:39:38 +0000
+++ doc/zorba/build_ubuntu.dox 2012-05-16 22:51:21 +0000
@@ -1,32 +1,86 @@
/**
-\page build_ubuntu Build Zorba on Ubuntu Linux
-
-\section bu_instructions Instructions
-
-For building Zorba on Ubuntu 11.04 (gcc version 4.5.2) folow the next steps:
-
-Install tools
-\code
+\page build_ubuntu Zorba Build Notes for Ubuntu/Debian Linux Users
+
+\section ubuntu_packages Ubuntu Packages
+
+Most of the tools and libraries you might need to build Zorba are
+available in the standard Ubuntu/Debian repositories. Here we list
+the packages that you should install to meet the requirements
+specified at \ref build_prerequisites.
+
+\subsection ubuntu_tools Tools
+To install the gcc compiler and CMake:
+\code
+sudo apt-get install build-essential cmake
+\endcode
+
+To install Bazaar (for building the latest Zorba source, or for
+building non-core modules):
+\code
+sudo apt-get install bzr
+\endcode
+
+\subsection ubuntu_libs Core Zorba Library Dependencies
+Required libraries for core Zorba:
+\code
+sudo apt-get install libxml2-dev
+\endcode
+
+Recommended libraries for core Zorba:
+\code
+sudo apt-get install libicu-dev libxerces-c-dev
+\endcode
+
+Optional libraries for core Zorba:
+\code
+sudo apt-get install libxslt-dev libcurl4-openssl-dev libxerces-c-dev
+\endcode
+
+SWIG and development requirements for Zorba language bindings:
+\code
+sudo apt-get install swig python-dev php5-dev ruby-dev openjdk-6-jdk libaccess-bridge-java-jni
+\endcode
+
+\subsection ubuntu_noncore_libs Non-Core Module Library Dependencies
+
+\subsubsection ubuntu_html HTML Module (data-converters package)
+\code
+sudo apt-get install libtidy-dev
+\endcode
+
+\subsubsection ubuntu_geos Geo Module
+
+To install libgeos for the Geo Module, first ensure that the version
+available for your distribution is at least 3.2.2:
+
+\code
+apt-cache show libgeos-dev | grep Version:
+\endcode
+
+If it is lower than 3.2.2, then you may add our PPA to your list of
+repositories to get a later version:
+
+\code
+sudo add-apt-repository ppa:juan457/zorba
sudo apt-get update
-sudo apt-get install subversion bzr build-essential cmake
-\endcode
-
-Optional
-\code
-sudo apt-get install valgrind
-\endcode
-
-Required libraries
-\code
-sudo apt-get install libxml2-dev libxslt-dev libicu-dev libcurl4-openssl-dev libtidy-dev libxerces-c-dev libz-dev
-\endcode
-
-
-\section bu_cclient Build C-Client library
-
-One option is to install the c-client lib from
-<a href=" http://launchpad.net/~juan457/+archive/zorba/+packages";>ppa:juan457/zorba</a>.
+\endcode
+
+In either case, install libgeos with:
+\code
+sudo apt-get install libgeos-dev
+\endcode
+
+\subsubsection ubuntu_imagemagick ImageMagick (for image modules)
+
+\code
+sudo apt-get install imagemagick libmagick++-dev libmagickwand-dev
+\endcode
+
+\subsubsection ubuntu_cclient Email Module
+
+We recommend using our PPA to install the C-Client module, as the
+Ubuntu packages are not built correctly at least on 64-bit platforms.
\code
sudo add-apt-repository ppa:juan457/zorba
@@ -34,8 +88,8 @@
sudo sudo apt-get install libc-client2007e libc-client2007e-dev
\endcode
-
-Or build it localy using the -I/usr/include/openssl -fPIC options:
+Or, you may build it yourself from source, so long as you force the
+-fPIC option:
\code
wget ftp://ftp.cac.washington.edu/imap/imap-2007e.tar.gz
@@ -45,96 +99,14 @@
make lnp EXTRACFLAGS="-I/usr/include/openssl -fPIC"
\endcode
-Note: To try again in case of errors, delete all, untar and make again, don't trust make clean.
-
-
-\section bu_imagemagick Install ImageMagick (for image modules)
-
-\code
-sudo apt-get install imagemagick libmagick++-dev
-\endcode
-
-
-
-\section bu_geos Install GEOS (for geo module)
-
-\code
-sudo apt-get install libgeos-dev
-\endcode
-
-If the version is older than 3.2.2 then there are two options:
-
-a. install it from <a href=" http://launchpad.net/~juan457/+archive/zorba/+packages";>ppa:juan457/zorba</a>.
-
-\code
-sudo add-apt-repository ppa:juan457/zorba
-sudo apt-get update
-sudo apt-get install libgeos libgeos-dev
-\endcode
-
-b. or compile it manually:
-
-\code
-sudo apt-get remove libgeos-dev
-svn checkout http://svn.osgeo.org/geos/trunk geos-svn
-cd geos-svn
-mkdir build
-cd build
-cmake ..
-make
-sudo make install
-\endcode
-
-
-
-\section bu_jansson Install Jansson (for JSON module) from ppa:petri/ppa
-
-\code
-sudo add-apt-repository ppa:petri/ppa
-sudo apt-get update
-sudo apt-get install libjansson4 libjansson-dev
-\endcode
-
-
-
-\section bu_build_zorba Get and build Zorba
-
-\code
-bzr branch lp:zorba
-cd zorba
-cmake -Doutdir=../zm -Dallmodules=1 -P modules/DownloadModules.cmake
-mkdir build
-cd build
-\endcode
-
-If all libraries were installed from ppa-s:
-\code
-cmake ..
-make
-\endcode
-
-Otherwise specify the location of c-client library:
-\code
-cmake -DCMAKE_BUILD_TYPE="Debug" -DCMAKE_PREFIX_PATH="~/imap-2007e/c-client" ../
-make
-\endcode
-
-
-
-\section bu_running_tests Run tests
-
-To run all Zorba tests:
-\code
-ctest
-\endcode
-
-
-
-\section bu_install Installation
-
-To install this version of Zorba:
-\code
-make install
-\endcode
-
-**/
+Note: In case of errors, delete all, untar and make again, don't trust
+make clean.
+
+If you do this, be sure to pass the path to the \"\c c-client\"
+subdirectory of your build to CMake when configuring Zorba, e.g.:
+\code
+cmake -DCMAKE_PREFIX_PATH="~/imap-2007e/c-client" ../
+make
+\endcode
+
+*/
=== added file 'doc/zorba/build_windows.dox'
--- doc/zorba/build_windows.dox 1970-01-01 00:00:00 +0000
+++ doc/zorba/build_windows.dox 2012-05-16 22:51:21 +0000
@@ -0,0 +1,109 @@
+/** \page build_windows Zorba Build Notes for Windows Users
+
+\section windows_requirements Requirements for Windows Builds
+- In order to compile Zorba on Windows,
+ you need a working Visual Studio (Express) installation (2008, 2010).
+- If you want to compile Zorba using Visual Studio 2008,
+ you will need to install the
+ <a href="http://www.microsoft.com/downloads/en/details.aspx?FamilyId=D466226B-8DAB-445F-A7B4-448B326C48E7"; target="_blank">Visual C++ 2008 Feature Pack Release</a>.
+- Make sure that the libraries required for Zorba
+ are built with the same version of Visual Studio that you use.
+- The Windows version of some required packages
+ are distributed by other web sites
+ (all of them are pointed to from the main package web sites
+ indicated in the \ref third_party_libs section):
+ - Libxml2 and Iconv: http://www.zlatkovic.com/libxml.en.html
+ - CURL: http://curl.haxx.se/download.html
+ - Libxslt: http://www.zlatkovic.com/pub/libxml/
+
+\section windows_autodll Zorba's Automatic DLL Detection Mechanism
+
+Zorba has in place an automatic DLL detection mechanism. This will
+try to automatically gather all the DLLs from the third party
+libraries and install them with Zorba when you do "make install" or
+you build the Visual Studio INSTALL project, to ensure that your
+installation will function regardless of the system or user's PATH.
+
+Zorba will search for the third party libraries in order in:
+-# all the directories pointed by the \c ZORBA_THIRD_PARTY_REQUIREMENTS
+ CMake configuration parameter
+ - e.g. if you have Xerces in "C:\tools" and IConv in "D:\tools",
+ you must configure your build as follows:
+ \code
+ cmake -D ZORBA_THIRD_PARTY_REQUIREMENTS=C:\tools;D:\tool {ZORBASRC}
+ \endcode
+ and Zorba will find all it needs from Xerces and Iconv.
+ (The directory names must contain "xerces" and "iconv" respectively, case insensitive)
+-# the <tt>Program Files</tt> directory on your system
+ - e.g. if you have cURL installed in "C:\Program Files\*cURL*",
+ Zorba will find all it needs from cURL.
+ (The directory must contain "curl", case insensitive.)
+-# all the paths in the \c PATH variable
+ - e.g. if you have "C:\tools\*ICU*\bin" in your path,
+ Zorba will find all it needs from ICU.
+ (The directory must only contain "icu".)
+
+The important thing to note here is that if you have any third-party
+requirements installed in unusual locations, you should use the CMake
+configuration parameter \c ZORBA_THIRD_PARTY_REQUIREMENTS instead of
+\c CMAKE_PREFIX_PATH to specify those locations when configuring
+Zorba.
+
+\section windows_nmake Building Zorba using a NMake Project
+-# Start a Visual Studio Command Line.
+-# Change the current working directory to the \c {ZORBABUILD} directory (as described in \ref create_build_dir).
+-# Execute CMake as follows:
+\code
+cmake -G "NMake Makefiles" {ZORBASRC}
+\endcode
+-# Now you are ready to call \"\c nmake\" from the \c {ZORBABUILD} directory.
+
+\section windows_visualstudio Building Zorba using a Visual Studio Project
+-# Start Visual Studio Command Line
+-# Change the current working directory into the \c {ZORBABUILD} directory (as described in \ref create_build_dir).
+-# Execute CMake with the appropriate CMake Visual Studio generator for your installation (see <tt>cmake --help</tt> for a list of generators). For instance,
+\code
+cmake -G "Visual Studio 10" {ZORBASRC}
+\endcode
+-# Start Visual Studio,
+ open the generated \c zorba.sln project file in \c {ZORBABUILD},
+ and build the \c ALL project.
+
+\section windows_https_support HTTPS support
+- In order to have SSL support in Zorba,
+ you have to use the CURL library that has SSL support.
+- There is one more thing:
+ in order to validate the server's certificate,
+ CURL+SSL needs to know about the root Certificates of Authenticity (CA):
+ a set of public keys freely available on the internet.
+- On Windows,
+ Zorba expects this file to be called \c cacert.pem
+ and searches for it in the current directory and system paths.
+- This \c cacert.pem file can be found on the curl page:
+ http://curl.haxx.se/docs/caextract.html .
+- In order to disable the validation of the server certificate,
+ the \c ZORBA_VERIFY_PEER_SSL_CERTIFICATE can be set to \c OFF
+ in cmake
+ (and actually this is the default value).
+ Set it to \c ON to validate server certificate chain
+ on every access to \c https:// .
+- To read more about CURL+SSL: http://curl.haxx.se/docs/sslcerts.html .
+- As an \c https test page,
+ we used for example the rss feed
+ fn:doc('https://www.npr.org/rss/rss.php?id=1001') .
+
+\section windows_noncore_modules Downloads for Non-Core Modules
+
+\subsection windows_imagemagick ImageMagick (image-processing modules)
+
+Just download the ImageMagick Windows binary from http://www.imagemagick.org/script/binary-releases.php and install it. We recommend the latest "Q8-windows-dll" version.
+
+\subsection windows_tidy HTML (data-converters modules)
+
+A binary installer for Tidy used to be available at:
+http://dev.int64.org/tidy.html, but as of this time of writing that
+site is down.
+
+
+
+*/
=== modified file 'doc/zorba/ft_intro.dox'
--- doc/zorba/ft_intro.dox 2012-04-24 12:39:38 +0000
+++ doc/zorba/ft_intro.dox 2012-05-16 22:51:21 +0000
@@ -5,9 +5,9 @@
specification.
Additional documentation:
- - \ref ft_stemmer
- - \ref ft_thesaurus
- - \ref ft_tokenizer
+- \ref ft_stemmer
+- \ref ft_thesaurus
+- \ref ft_tokenizer
\section ft_unimplemented Unimplemented Features
@@ -16,11 +16,11 @@
implemented.
The features that are not (completely) implemented are:
- - The <a href="http://www.w3.org/TR/xpath-full-text-10/#ftignoreoption";>Ignore Option</a>
- (bug <a href="https://bugs.launchpad.net/zorba/+bug/sf-3187470";>3187470</a>).
- - <a href="http://www.w3.org/TR/xpath-full-text-10/#section-score-variables";>Score Variables</a>
- and <a href="http://www.w3.org/TR/xpath-full-text-10/#section-using-weights";>Using Weights Within a Scored FTContainsExpr</a>
- (bug <a href="https://bugs.launchpad.net/zorba/+bug/sf-3187462";>3187462</a>).
+- The <a href="http://www.w3.org/TR/xpath-full-text-10/#ftignoreoption";>Ignore Option</a>
+ (bug <a href="https://bugs.launchpad.net/zorba/+bug/866924";>866924</a>).
+- <a href="http://www.w3.org/TR/xpath-full-text-10/#section-score-variables";>Score Variables</a>
+ and <a href="http://www.w3.org/TR/xpath-full-text-10/#section-using-weights";>Using Weights Within a Scored FTContainsExpr</a>
+ (bug <a href="https://bugs.launchpad.net/zorba/+bug/866923";>866923</a>).
*/
/* vim:set et sw=2 ts=2: */
=== modified file 'doc/zorba/ft_stemmer.dox'
--- doc/zorba/ft_stemmer.dox 2012-04-24 12:39:38 +0000
+++ doc/zorba/ft_stemmer.dox 2012-05-16 22:51:21 +0000
@@ -56,7 +56,12 @@
public:
typedef /* implementation-defined */ ptr;
+ struct Properties {
+ char const *uri;
+ };
+
virtual void destroy() const = 0;
+ virtual void properties( Properties *result ) const = 0;
virtual void stem( String const &word, locale::iso639_1::type lang, String *result ) const = 0;
protected:
virtual ~Stemmer();
@@ -89,6 +94,8 @@
Note that \c result should always be set to something.
If your stemmer doesn't know how to stem the given word,
you should set \c result to \c word.
+You also need to implement the \c properties() function
+and set the identifying URI of your stemmer.
A very simple stemmer
that stems the word "foobar" to "foo"
@@ -98,6 +105,7 @@
class MyStemmer : public Stemmer {
public:
void destroy() const;
+ void properties( Properties *result ) const;
void stem( String const &word, locale::iso639_1::type lang, String *result ) const;
private:
MyStemmer();
@@ -108,6 +116,10 @@
// Do nothing since we statically allocate a singleton instance of our stemmer.
}
+void MyStemmer::properties( Properties *props ) const {
+ props->uri = "http://my.example.com/zorba/full-text/stemmer";;
+}
+
void MyStemmer::stem( String const &word, locale::iso639_1::type lang, String *result ) const {
if ( word == "foobar" )
*result = "foo";
@@ -120,7 +132,6 @@
or a dictionary look-up
to stem many words,
of course.
-
Although not used in this simple example,
\c lang can be used to allow a single stemmer instance
to stem words in more than one language.
@@ -135,16 +146,24 @@
class StemmerProvider {
public:
virtual ~StemmerProvider();
- virtual Stemmer::ptr getStemmer( locale::iso639_1::type lang ) const = 0;
+ virtual bool getStemmer( locale::iso639_1::type lang, Stemmer::ptr *s = 0 ) const = 0;
};
\endcode
+The \c getStemmer() function should return \c true
+only if it can provide a \c Stemmer
+for the given language; \c false otherwise.
+If the \c Stemmer::ptr argument is \c null,
+the caller wants to check only whether the provider
+can provide a stemmer for the given language
+and doesn't want a \c Stemmer instance created or returned.
+
A simple \c StemmerProvider for our simple stemmer can be implemented as:
\code
class MyStemmerProvider : public StemmerProvider {
public:
- Stemmer::ptr getStemmer( locale::iso639_1::type lang ) const;
+ bool getStemmer( locale::iso639_1::type lang Stemmer::ptr *s = 0 ) const;
};
Stemmer::ptr MyStemmerProvider::getStemmer( locale::iso639_1::type lang ) const {
@@ -154,15 +173,14 @@
case iso639_1::en:
case iso639_1::unknown: // Handle "unknown" language since, in many cases, the language is not known.
result.reset( &stemmer );
- break;
+ return true;
default:
//
- // We have no stemmer for the given language: leave the result as null to indicate this.
+ // We have no stemmer for the given language: return false.
// Zorba will then use the built-in stemmer for the given language.
//
- break;
+ return false;
}
- resturn std::move( result );
}
\endcode
=== modified file 'doc/zorba/ft_thesaurus.dox'
--- doc/zorba/ft_thesaurus.dox 2012-04-24 12:39:38 +0000
+++ doc/zorba/ft_thesaurus.dox 2012-05-16 22:51:21 +0000
@@ -44,16 +44,16 @@
To download and install the WordNet database on a Unix-like system,
follow these steps:
- -# Download the WordNet database from
- <a href="http://wordnet.princeton.edu/wordnet/download/";>here</a>.
- All you really need are just the database files
- (<code>WNdb-3.0.tar.gz</code>).
- -# Un-gzip and untar the files.
- This will result in a directory dict containing the database files.
- -# Move the dict directory somewhere of your choosing,
- e.g., <code>/usr/local/wordnet-3.0/dict</code>.
- -# Compile the \c dict directory into a Zorba-compatible binary thesaurus
- as described below.
+-# Download the WordNet database from
+ <a href="http://wordnet.princeton.edu/wordnet/download/";>here</a>.
+ All you really need are just the database files
+ (<code>WNdb-3.0.tar.gz</code>).
+-# Un-gzip and untar the files.
+ This will result in a directory dict containing the database files.
+-# Move the dict directory somewhere of your choosing,
+ e.g., <code>/usr/local/wordnet-3.0/dict</code>.
+-# Compile the \c dict directory into a Zorba-compatible binary thesaurus
+ as described below.
To compile the WordNet database files,
use the \c zt-wn-compile script
@@ -65,12 +65,12 @@
zt-wn-compile [-v] wordnet_dict_dir [thesaurus_file]
\endcode
- - The \c -v option specifies verbose output.
- - The \e wordnet_dict_dir specifies the full path
- of the WordNet \c dict directory.
- - The \e thesaurus_file specifies the name of the resulting binary file.
- If none is given, it defaults to \c wordnet-en.zth
- ("en" for English and "zth" for "Zorba Thesaurus file").
+- The \c -v option specifies verbose output.
+- The \e wordnet_dict_dir specifies the full path
+ of the WordNet \c dict directory.
+- The \e thesaurus_file specifies the name of the resulting binary file.
+ If none is given, it defaults to \c wordnet-en.zth
+ ("en" for English and "zth" for "Zorba Thesaurus file").
For example:
@@ -78,33 +78,39 @@
zt-wn-compile -v /usr/local/wordnet-3.0/dict
\endcode
-Move the \c wordnet-en.zth file to a location of your choosing.
+To install the \c wordnet-en.zth file,
+move it onto Zorba's <i>library path</i>:
+
+\code
+LIB_PATH/edu/princeton/wordnet/wordnet-en.zth
+\endcode
\subsection ft_thesaurus_precompiled Downloading a Precompiled WordNet Database
Alternatively,
-you can download a precompiled WordNet database from
+you can download a precompiled, little-endian (Intel) CPU WordNet database from
<a href="http://www.zorba-xquery.com/downloads/WordNet-3.0/wordnet-en.zip";>here</a>.
\section ft_thesaurus_mappings Thesauri Mappings
In order to use thesauri,
-you need to specify where they are to the Zorba engine
-via one or more thesaurus <i>mappings</i>.
-A <i>mapping</i> maps a symbolic URI to URI for an actual thesaurus.
+you need to specify what symbolic URI(s) <i>map</i>
+to what thesauri.
A mapping is of the form:
-<i>from_uri</i><code>:=</code><b>[</b><i>implementation</i><code>|</code><b>]</b><i>to_uri</i>
+<i>from_uri</i><code>:=</code><i>implementation-scheme</i><code>:</code><i>to_uri</i>
For example:
\code
-http://wordnet.princeton.edu:=wordnet|/usr/local/zorba/thesauri/wordnet-en.zth
+http://wordnet.princeton.edu:=wordnet://wordnet.princeton.edu
\endcode
says that the symbolic URI \c http://wordnet.princeton.edu
maps to the WordNet implementation
-having a database file at the given path.
+having a database file at the given sub-path
+\c edu/princeton/wordnet
+on Zorba's library path.
Once a mapping is established for a symbolic URI,
it can be used in a query:
@@ -114,13 +120,8 @@
using thesaurus at "http://wordnet.princeton.edu";
\endcode
-If the \e implementation is omitted,
-it defaults to \c wordnet.
As a special-case,
-the \e from_uri can be \c default or
-\code
-##default
-\endcode
+the \e from_uri can be \c default or \c ##default
to allow for specifying the default thesaurus
as was done for the first example on this page.
@@ -130,7 +131,7 @@
use one or more –thesaurus options:
\code
-zorba --thesaurus default:=/usr/local/zorba/thesauri/wordnet-en.zth ...
+zorba --thesaurus default:=wordnet://wordnet.princeton.edu ...
\endcode
\section ft_thesaurus_rels Thesaurus Relationships
@@ -423,25 +424,26 @@
If no levels are specified in a query,
Zorba defaults the WordNet implementation to be 2 levels.
-The rationale can be found
-<a href="http://www.w3.org/Bugs/Public/show_bug.cgi?id=11444";>here</a>.
+(The rationale can be found
+<a href="http://www.w3.org/Bugs/Public/show_bug.cgi?id=11444";>here</a>.)
\section ft_thesaurus_providing Providing Your Own Thesaurus
Using the Zorba C++ API,
you can provide your own thesaurus
-by deriving from three classes:
+by deriving from four classes:
\c Thesaurus,
\c Thesaurus::iterator,
+\c ThesaurusProvider,
and
-\c ThesaurusProvider.
+\c URLResolver.
\subsection ft_class_thesaurus The Thesaurus Class
The \c Thesaurus class is:
\code
-class Thesaurus : public Resource {
+class Thesaurus {
public:
typedef /* implementation-defined */ ptr;
typedef /* implementation-defined */ range_type;
@@ -457,15 +459,15 @@
virtual iterator::ptr lookup( String const &phrase, String const &relationship, range_type at_least, range_type at_most ) const = 0;
- virtual void destroy() const = 0; // interited from Resource
+ virtual void destroy() const = 0;
protected:
virtual ~Thesaurus();
};
\endcode
-For details about the \c ptr type,
-the \c destroy() function,
-and why the destructor is \c protected,
+For details about the \c ptr types,
+the \c destroy() functions,
+and why the destructors are \c protected,
see the \ref memory_management document.
To implement the \c Thesaurus
@@ -482,18 +484,19 @@
</tr>
<tr>
<td>\c at_least</td>
- <td>The The minimum number of levels within the thesaurus to be traversed.</td>
+ <td>The minimum number of levels within the thesaurus to be traversed.</td>
</tr>
<tr>
<td>\c at_most</td>
- <td>The The maximum number of levels within the thesaurus to be traversed.</td>
+ <td>The maximum number of levels within the thesaurus to be traversed.</td>
</tr>
</table>
The \c lookup() function returns a pointer to an \c iterator
that is used to iterate over the phrase's synonyms.
-
-A very simple thesaurus
+You also need to implement an \c iterator.
+A very simple \c Thesaurus
+and its \c iterator
can be implemented as:
\code
@@ -505,53 +508,49 @@
//
// Define a simple thesaurus data structure as a map from a phrase to a list of its synonyms.
//
- typedef std::list<String> synonyms_t;
- typedef std::map<String,synonyms_t const*> thesaurus_t;
+ typedef std::list<String> synonyms_type;
+ typedef std::map<String,synonyms_type const*> thesaurus_data_type;
- static thesaurus_t const& get_thesaurus();
+ static thesaurus_data_type const& get_thesaurus_data();
class iterator : public Thesaurus::iterator {
public:
- iterator( synonyms_t const &s ) : synonyms_( s ), i_( s.begin() ) { }
+ iterator( synonyms_type const &s ) : synonyms_( s ), i_( s.begin() ) { }
void destroy();
bool next( String *synonym );
private:
- synonyms_t const &synonyms_; // synonyms to iterate over
- synonyms_t::const_iterator i_; // current iterator position
+ synonyms_type const &synonyms_; // synonyms to iterate over
+ synonyms_type::const_iterator i_; // current iterator position
};
};
void MyThesaurus::destroy() const {
- // Do nothing since we statically allocate a singleton instance of our thesaurus.
+ // Do nothing since we statically allocate a singleton instance of our Thesaurus.
}
-MyThesaurus::thesaurus_t const& MyThesaurus::get_thesaurus() {
- static thesaurus_t thesaurus;
- if ( thesaurus.empty() ) {
- //
- // Construct a thesaurus "by hand" for this example. A real thesaurus would probably
- // be read from disk.
- //
+MyThesaurus::thesaurus_data_type const& MyThesaurus::get_thesaurus_data() {
+ static thesaurus_data_type thesaurus_data;
+ if ( thesaurus_data.empty() ) {
+ //
+ // Construct thesaurus data "by hand" for this example. A real thesaurus would probably be read from disk.
// Note that every list of synonyms must always include the original phrase.
//
- static synonyms_t synonyms;
+ static synonyms_type synonyms;
synonyms.push_back( "foo" );
synonyms.push_back( "foobar" );
- thesaurus[ "foo" ] = &synonyms;
- thesaurus[ "foobar" ] = &synonyms;
+ thesaurus_data[ "foo" ] = &synonyms;
+ thesaurus_data[ "foobar" ] = &synonyms;
}
- return thesaurus;
+ return thesaurus_data;
}
-\endcode
-\code
MyThesaurus::iterator::ptr MyThesaurus::lookup( String const &phrase, String const &relationship,
range_type at_least, range_type at_most ) const {
- static thesaurus_t const &thesaurus = get_thesaurus();
- thesaurus_t::const_iterator const i = thesaurus.find( phrase );
+ static thesaurus_data_type const &thesaurus_data = get_thesaurus_data();
+ thesaurus_data_type::const_iterator const entry = thesaurus_data.find( phrase );
iterator::ptr result;
- if ( i != thesaurus.end() )
- result.reset( new iterator( *i->second ) );
+ if ( entry != thesaurus_data.end() )
+ result.reset( new iterator( *entry->second ) );
return std::move( result );
}
@@ -572,13 +571,71 @@
A real thesaurus would load a large number of synonyms,
of course.
+\subsection ft_class_thesaurus_provider The ThesaurusProvider Class
+
+The \c ThesaurusProvider class is:
+
+\code
+class ThesaurusProvider : public Resource {
+public:
+ typedef /* implementation-defined */ ptr;
+
+ virtual bool getThesaurus( locale::iso639_1::type lang, Thesaurus::ptr *thesaurus = 0 ) const = 0;
+ void destroy() const; // inherited from Resource
+};
+\endcode
+
+To implement a \c ThesaurusProvider,
+you need to implement the \c getThesaurus() function where:
+
+<table>
+ <tr>
+ <td>\c lang</td>
+ <td>The desired language of the thesaurus.</td>
+ </tr>
+ <tr>
+ <td>\c thesaurus</td>
+ <td>If not \c null, set to point to a thesaurus for \c lang.</td>
+ </tr>
+</table>
+
+The \c getThesaurus() function returns \c true
+only if it can provide a thesaurus for the given language.
+Continuing with the example,
+a very simple \c ThesaurusProvider
+can be implemented as:
+
+\code
+class MyThesaurusProvider : pulic ThesaurusProvider {
+public:
+ void destroy() const;
+ bool getThesaurus( iso639_1::type lang, Thesaurus::ptr* = 0 ) const;
+};
+
+void MyThesaurusProvider::destroy() const {
+ // Do nothing since we statically allocate a singleton instance of our ThesaurusProvider.
+}
+
+bool MyThesaurusProvider::getThesaurus( iso639_1::type lang, Thesaurus::ptr *result ) const {
+ //
+ // Since our tiny thesaurus contains only universally known words, we don't bother checking lang
+ // and always return true.
+ //
+ static MyThesaurus thesaurus;
+ if ( result )
+ result->reset( &thesaurus );
+ return true;
+}
+\endcode
+
\subsection ft_class_thesaurus_resolver A Thesaurus URL Resolver Class
-In addition to a \c Thesaurus,
+In addition to a \c Thesaurus
+and \c ThesaurusProvider,
you must also implement a "thesaurus resolver" class
that,
-given a URL and a language,
-provides a \c Thesaurus for that language.
+given a URI,
+provides a \c ThesaurusProvider for that URI.
A simple \c ThesaurusURLResolver
for our simple thesaurus can be implemented as:
@@ -591,23 +648,12 @@
String const url_;
};
-Resource*
-ThesaurusURLResolver::resolveURL( String const &url, EntityData const *data ) const {
- ThesaurusEntityData const *const t_data = dynamic_cast<ThesaurusEntityData const*>( data );
- assert( t_data );
- static MyThesaurus thesaurus;
- if ( url == url_ )
- switch ( t_data->getLanguage() ) {
- case locale::iso639_1::en:
- case locale::iso639_1::unknown:
- //
- // Here, we could test to ensure that the language of our thesaurus matches the
- // language sought, but in our case, we want our thesaurus to be used for all
- // languages since "foo" and "foobar" are universal.
- //
- default:
- return &thesaurus;
- }
+Resource* ThesaurusURLResolver::resolveURL( String const &url, EntityData const *data ) const {
+ if ( data->getKind() == EntityData::THESAURUS )
+ static MyThesaurusProvider provider;
+ if ( uri == uri_ )
+ return &provider;
+ }
return 0;
}
\endcode
=== modified file 'doc/zorba/ft_tokenizer.dox'
--- doc/zorba/ft_tokenizer.dox 2012-04-24 12:39:38 +0000
+++ doc/zorba/ft_tokenizer.dox 2012-05-16 22:51:21 +0000
@@ -5,14 +5,25 @@
The Zorba XQuery processor implements the
<a href="http://www.w3.org/TR/xpath-full-text-10/";>XQuery and XPath Full Text 1.0</a>
specification that, among other things,
-tokenizes a string into a sequence of tokens.
-See
-<a href="http://www.w3.org/TR/xpath-full-text-10/#TokenizationSec";>Tokenization</a>.
-
-The initial implementation of the toknenizer
-uses the one provided by the
-<a href="http://site.icu-project.org/";>ICU library</a>.
-However, you can provide your own tokenizer instead.
+<a ref="http://www.w3.org/TR/xpath-full-text-10/#TokenizationSec";>tokenizes</a>
+a string into a sequence of tokens.
+
+\section ft_tokenizer_tokization Tokenization
+
+Using the
+<a href="http://site.icu-project.org/";>ICU library</a>,
+Zorba's implementation of tokenization
+considers only alpha-numeric sequences of characters to be part of a token;
+whitespace and punctuation characters are not
+and separate tokens.
+However, alpha-numeric sequences matching the regular expression
+<code>[0-9][.,][0-9]</code>
+are retained as part of a token, e.g.:
+"98.6" and "1,432.58" are tokens.
+
+Alternatively,
+you can implement your own tokenizer
+by deriving from the \c Tokenizer class.
\section ft_class_tokenizer The Tokenizer Class
@@ -36,33 +47,43 @@
class Callback {
public:
- typedef Tokenizer::size_type size_type;;
+ typedef Tokenizer::size_type size_type;
virtual ~Callback();
- virtual void operator()( char const *utf8_s, size_type utf8_len,
- size_type token_no, size_type sent_no, size_type para_no,
- void *payload = 0 ) = 0;
- };
-
- enum ElementTraceOptions {
- trace_none = 0x0, // Trace no elements.
- trace_begin = 0x1, // Trace the beginning of elements.
- trace_end = 0x2 // Trace the ending of elements.
- };
+ virtual void token( char const *utf8_s, size_type utf8_len, locale::iso639_1::type lang,
+ size_type token_no, size_type sent_no, size_type para_no,
+ Item const *item = 0 ) = 0;
+ };
+
+ struct Properties {
+ typedef std::vector<locale::iso639_1::type> languages_type;
+
+ bool comments_separate_tokens;
+ bool elements_separate_tokens;
+ bool processing_instructions_separate_tokens;
+ languages_type languages;
+ char const *uri;
+ };
+
+ virtual void properties( Properties *result ) const = 0;
virtual void destroy() const = 0;
- virtual void element( Item const &qname, int trace_options );
Numbers& numbers();
Numbers const& numbers() const;
- int trace_options() const;
-
- virtual void tokenize( char const *utf8_s, size_type utf8_len, locale::iso639_1::type lang,
- bool wildcards, Callback &callback, void *payload = 0 ) = 0;
+
+ void tokenize_node( Item const &node, locale::iso639_1::type lang, Callback &callback );
+
+ virtual void tokenize_string( char const *utf8_s, size_type utf8_len, locale::iso639_1::type lang,
+ bool wildcards, Callback &callback, Item const *item = 0 ) = 0;
protected:
- Tokenizer( Numbers&, int trace_options = trace_none );
+ Tokenizer( Numbers& );
virtual ~Tokenizer();
+
+ bool find_lang_attribute( Item const&, locale::iso639_1::type *lang );
+ virtual void item( Item const&, bool entering );
+ virtual void tokenize_node_impl( Item const&, locale::iso639_1::type, Callback&, bool tokenize_acp );
};
\endcode
@@ -76,8 +97,8 @@
It simply keeps track of the current
token, sentence, and paragraph numbers.
-To implement the \c Tokenizer,
-you need to implement the \c %tokenize() function where:
+To implement a \c Tokenizer,
+you need to implement the \c %tokenize_string() function where:
<table>
<tr>
@@ -115,9 +136,13 @@
</td>
</tr>
<tr>
- <td>\c payload</td>
+ <td>\c item</td>
<td>
- Optional implementation-defined data.
+ The \c Item whence this token came.
+ If the token occurred within an element,
+ the \c Item is the text node.
+ If the token occurred within an attribute,
+ the \c Item is the attribute node.
</td>
</tr>
</table>
@@ -127,21 +152,30 @@
However,
the things a tokenizer should take into consideration include:
- - Detecting sentence termination ('.', '?', and '!' characters).
- - Handling floating-point numbers with possible thousands separators
- in US and European formats, e.g. "98.7", "98,7", "10,000", etc.
- - Distinguishing '.' used as a sentence terminator
- from '.' used as a decimal point.
- - Handling apostrophies, e.g., "men's".
- - Handling acronyms, e.g., "AT&T".
-
-\subsection ft_paragraphs Paragraphs
+- Detecting sentence termination ('.', '?', and '!' characters).
+- Handling floating-point numbers with possible thousands separators
+ in US and European formats, e.g. "98.7", "98,7", "10,000", etc.
+- Distinguishing '.' used as a sentence terminator
+ from '.' used as a decimal point.
+- Handling apostrophies, e.g., "men's".
+- Handling acronyms, e.g., "AT&T".
+
+The task of iterating over an XML element's child nodes
+is done by \c tokenize_node_impl().
+Its default implementation
+treats XML elements, comments, and processing instructions
+as token separators.
+(See \ref ft_tokenizer_properties.)
+If you want to change that,
+you need to override \c tokenize_node_impl().
+
+\subsection ft_tokenizer_paragraphs Paragraphs
By default,
Zorba increments the current paragraph number once
for each XML element encountered.
However,
-this doens't work well for mixed content.
+this doesn't work well for mixed content.
For example, in the XHTML:
\code
<p>The <em>best</em> thing ever!</p>
@@ -150,31 +184,65 @@
but Zorba will consider that 3 paragraphs by default.
Your tokenizer can take control over when the paragraph number is incremented
-by passing the bitwise-or
-of the \c ElementTraceOptions values
-to the constructor
-and overriding the \c element() function.
-The \c element() function is passed the QName of the current XML element
-and (depending on the initial value passed to the constructor)
-one of \c trace_begin or \c trace_end.
-Note that this function is called
-only if the trace options value
-passed to the constructor
-was non-zero.
+by overriding the \c item() function.
+The \c item() function is passed the \c Item of the current XML element
+and whether the item is being entered or exited.
For example,
-the \c element() function for tokenizing XHTML
+the \c item() function for tokenizing XHTML
would be along the lines of:
\code
-void MyTokenizer::element( Item const &qname, int trace_options ) {
- if ( trace_options & trace_end )
- return;
- String const name( qname.getLocalName() );
- if ( /* qname is an XHTML block-level element */ )
- ++numbers().para;
+void MyTokenizer::item( Item const &item, bool entering ) {
+ if ( entering && item.isNode() && item.getNodeKind() == store::StoreConsts::elementNode ) {
+ Item qname;
+ item.getNodeName( qname );
+ if ( /* qname matches an XHTML block-level element's name */ )
+ ++numbers().para;
}
\endcode
+\subsection ft_tokenizer_properties Properties
+
+To implement a \c Tokenizer,
+you need also to implement the \c %properties() function
+that fills in the \c Properties struct where:
+
+<table>
+ <tr>
+ <td>\c comments_separate_tokens</td>
+ <td>
+ If \c true, XML comments separate tokens. For example,
+ <code>net<!-- -->work</code> would be 2 tokens instead of 1.
+ </td>
+ </tr>
+ <tr>
+ <td>\c elements_separate_tokens</td>
+ <td>
+ If \c true, XML elements separate tokens. For example,
+ <code><b>B</b>old</code> would be 2 tokens instead of 1.
+ </td>
+ </tr>
+ <tr>
+ <td>\c processing_instructions_separate_tokens</td>
+ <td>
+ If \c true, XML processing instructions separate tokens. For example,
+ <code>net<?PI pi?>work</code> would be 2 tokens instead of 1.
+ </td>
+ </tr>
+ <tr>
+ <td>\c languages</td>
+ <td>
+ The list of languages supported by the tokenizer.
+ </td>
+ </tr>
+ <tr>
+ <td>\c uri</td>
+ <td>
+ The URI that uniquely identifies the %Tokenizer.
+ </td>
+ </tr>
+</table>
+
\section ft_class_tokenizer_provider The TokenizerProviderClass
In addition to a \c Tokenizer,
@@ -185,20 +253,51 @@
class TokenizerProvider {
public:
virtual ~TokenizerProvider();
- virtual Tokenizer::ptr getTokenizer( locale::iso639_1::type lang, Tokenizer::Numbers &numbers ) const = 0;
+ virtual bool getTokenizer( locale::iso639_1::type lang, Tokenizer::Numbers *numbers = 0, Tokenizer::ptr* = 0 ) const = 0;
};
\endcode
+Specifically, you need to implement the \c getTokenizer() function where:
+
+<table>
+ <tr>
+ <td>\c lang</td>
+ <td>The language to tokenize.</td>
+ </tr>
+ <tr>
+ <td>\c num</td>
+ <td>
+ The \c Numbers to use.
+ If \c null,
+ \a t is not set.
+ </td>
+ </tr>
+ <tr>
+ <td>\c t</td>
+ <td>
+ If not \c null,
+ set to point to a Tokenizer for \a lang.
+ </td>
+ </tr>
+</table>
+
A simple \c TokenizerProvider for our tokenizer can be implemented as:
\code
class MyTokenizerProvider : public TokenizerProvider {
public:
- Tokenizer::ptr getTokenizer( locale::iso639_1::type lang ) const;
+ getTokenizer( locale::iso639_1::type lang, Tokenizer::Numbers* = 0, Tokenizer::ptr* = 0 ) const;
};
-Tokenizer::ptr MyTokenizerProvider::getTokenizer( locale::iso639_1::type lang const {
- return Tokenizer::ptr( new MyTokenizer );
+bool MyTokenizerProvider::getTokenizer( locale::iso639_1::type lang, Tokenizer::Numbers *num, Tokenizer::ptr *t ) const {
+ switch ( lang ) {
+ case iso639_1::en:
+ if ( num && t )
+ t->reset( new MyTokenizer );
+ return true;
+ default:
+ return false;
+ }
}
\endcode
=== modified file 'doc/zorba/indexpage.dox.in'
--- doc/zorba/indexpage.dox.in 2012-04-24 12:39:38 +0000
+++ doc/zorba/indexpage.dox.in 2012-05-16 22:51:21 +0000
@@ -1,9 +1,7 @@
/*! \mainpage Zorba ${ZORBA_MAJOR_NUMBER}.${ZORBA_MINOR_NUMBER}.${ZORBA_PATCH_NUMBER} Documentation
-\section installbuild Installation and Build
- - \ref installation
+\section installbuild Building Zorba
- \ref build
- - \ref build_ubuntu
- \ref php_ubuntu_tutorial
- \ref php_windows_tutorial
- \ref python_windows_tutorial
=== removed file 'doc/zorba/installation.dox'
--- doc/zorba/installation.dox 2012-04-24 12:39:38 +0000
+++ doc/zorba/installation.dox 1970-01-01 00:00:00 +0000
@@ -1,96 +0,0 @@
-/** \page installation Installation
-
-\section SourceInstallation Installing the Source Distribution
-
-The Zorba source is available as tar.gz and zip files.
-The Zorba sources can be installed by extracting the compressed files
-(e.g., <tt>tar xvfz zorba-</tt><em>version</em><tt>.tar.gz</tt>).
-This creates a <tt>zorba-</tt><em>version</em> directory
-in your current working directory.
-
-To build Zorba,
-please follow the build instructions (see \ref buildingzorba).
-
-\section Installing Installing
-
-Once you have successfully built Zorba using the "Unix Makefile" generator
-(see \ref SourceInstallation),
-you can execute <tt>make install</tt>
-from within the build directory
-to install Zorba.
-This will install Zorba
-into the path specified by the \c CMAKE_INSTALL_PREFIX variable.
-
-Note that for setting the install prefix,
-you can use the \c CMAKE_INSTALL_PREFIX variable.
-For example,
-in order to install Zorba in <tt>/opt</tt>
-use <tt>cmake -D CMAKE_INSTALL_PREFIX=/opt </tt><em>zorba source directory</em>.
-(Note that while CMake-generated Makefiles support the \c DESTDIR variable,
-this will NOT work with Zorba as certain installation paths are hard-coded
-in the binary files.)
-
-There is also a <tt>make uninstall</tt> command available
-(also to be called from the build directory).
-
-Alternatively,
-you can use one of Zorba's binary distributions
-if it is available for your platform.
-For example,
-download and decompress the tar-ball
-for Mac OS X (e.g., <tt>zorba-</tt><em>version</em><tt>-Darwin-i386.tar.gz</tt>).
-
-For both installations,
-you will need to add Zorba's bin directory
-(e.g., \c CMAKE_INSTALL_PREFIX/bin or
-<tt>zorba-0.9.2541-Darwin-i386/bin</tt>)
-to your \c PATH environment variable.
-
-Additionally,
-you will need to add the corresponding library search path
-(e.g., \c LD_LIBRARY_PATH on Linux,
-\c DYLD_LIBRARY_PATH on Mac,
-and \c PATH on Windows)
-to point to the lib directory
-(e.g., \c CMAKE_INSTALL_PREFIX/lib or
-<tt>zorba-</tt><em>version</em><tt>-Darwin-i386/lib</tt>).
-Also,
-please make sure that the directories of the required libraries
-(see \ref build)
-are present in your library search path.
-
-After that,
-try typing
-<tt>%zorba -q "1+1"</tt>
-and you should see
-\code
-<?xml version="1.0" encoding="UTF-8"?>
-2
-\endcode
-
-as a result.
-Congratulations -- you're done and ready to use Zorba.
-Otherwise, please consult the zorba-users mailing list for help.
-
-For programming against the Zorba library,
-we refer to the API examples
-provided with the source and binary distribution
-(see \link simple.cpp \endlink for an example).
-
-In order to make the installation process easier,
-we also provide graphical installers
-for Windows and Mac OS X.
-
-\section UbuntuInstallation Additional notes for Ubuntu systems installations
-
-The recommended way of installing the Zorba library on an Ubuntu system
-is to add the lib directory as a \c .conf file in \c /etc/ld.so.conf
-and then run <tt>sudo ldconfig</tt>.
-For a personal (non-root non-system wide) installation,
-employing \c LD_LIBRARY_PATH is the only valid approach,
-but it needs to be done in the shell's \c .rc file
-not in the \c .profile file.
-Additional information can be found here:
-<a href="https://edge.launchpad.net/ubuntu/+bug/366728";>LD_LIBRARY_PATH not loading from .profile nor /etc/environment</a>.
-
-*/
=== modified file 'doc/zorba/modules_authoring_2.dox'
--- doc/zorba/modules_authoring_2.dox 2012-04-24 12:39:38 +0000
+++ doc/zorba/modules_authoring_2.dox 2012-05-16 22:51:21 +0000
@@ -318,6 +318,21 @@
directory will be set in a CMake variable by the
<tt>FIND_PACKAGE()</tt> command.
+Important note: <tt>DECLARE_ZORBA_MODULE</tt> discovers all
+<tt>.cpp</tt> files in the source directory automatically, using a
+glob pattern (i.e., <tt>*.cpp</tt>). This is convenient, but it does
+have a hidden downside: If you add new <tt>.cpp</tt> files to this
+directory, CMake has no way of knowing that you have done so, and so
+those files will not automatically get built. This can lead to strange
+compilation or runtime errors. Fortunately, there is a simple
+solution: Whenever you add new <tt>.cpp</tt> files for a module,
+always remember to immediately re-run <tt>cmake</tt> in your build
+directory. This will cause the glob pattern to be re-run and will pick
+up any new source code files. Likewise, if you delete any
+<tt>.cpp</tt> files from your source directory, you will need to
+re-run <tt>cmake</tt> for the same reason - although at least in that
+case you will get a clear error message if you forget to do so.
+
\section mod_author_declare_synopsis Zorba Module CMake Macros: Summary
For reference, here is the complete set of options for all macros
=== modified file 'include/zorba/locale.h'
--- include/zorba/locale.h 2012-04-24 12:39:38 +0000
+++ include/zorba/locale.h 2012-05-16 22:51:21 +0000
@@ -22,24 +22,198 @@
///////////////////////////////////////////////////////////////////////////
+ /**
+ * Defines constants for all ISO 639-1 language codes.
+ */
namespace iso639_1 {
enum type {
unknown,
- da, // Danish
- de, // German
- en, // English
- es, // Spanish
- fi, // Finnish
- fr, // French
- hu, // Hungarian
- it, // Italian
- nl, // Dutch
- no, // Norwegian
- pt, // Portuguese
- ro, // Romanian
- ru, // Russian
- sv, // Swedish
- tr, // Turkish
+ aa, ///< Afar
+ ab, ///< Abkhazian
+ ae, ///< Avestan
+ af, ///< Afrikaans
+ ak, ///< Akan
+ am, ///< Amharic
+ an, ///< Aragonese
+ ar, ///< Arabic
+ as, ///< Assamese
+ av, ///< Avaric
+ ay, ///< Aymara
+ az, ///< Azerbaijani
+ ba, ///< Bashkir
+ be, ///< Byelorussian
+ bg, ///< Bulgarian
+ bh, ///< Bihari
+ bi, ///< Bislama
+ bm, ///< Bambara
+ bn, ///< Bengali; Bangla
+ bo, ///< Tibetan
+ br, ///< Breton
+ bs, ///< Bosnian
+ ca, ///< Catalan
+ ce, ///< Chechen
+ ch, ///< Chamorro
+ co, ///< Corsican
+ cr, ///< Cree
+ cs, ///< Czech
+ cu, ///< Church Slavic; Church Slavonic
+ cv, ///< Chuvash
+ cy, ///< Welsh
+ da, ///< Danish
+ de, ///< German
+ dv, ///< Divehi
+ dz, ///< Bhutani
+ ee, ///< Ewe
+ el, ///< Greek
+ en, ///< English
+ eo, ///< Esperanto
+ es, ///< Spanish
+ et, ///< Estonian
+ eu, ///< Basque
+ fa, ///< Persian
+ ff, ///< Fulah
+ fi, ///< Finnish
+ fj, ///< Fiji
+ fo, ///< Faroese
+ fr, ///< French
+ fy, ///< Frisian
+ ga, ///< Irish
+ gd, ///< Scots Gaelic
+ gl, ///< Galician
+ gn, ///< Guarani
+ gu, ///< Gujarati
+ gv, ///< Manx
+ ha, ///< Hausa
+ he, ///< Hebrew (formerly iw)
+ hi, ///< Hindi
+ ho, ///< Hiri Motu
+ hr, ///< Croatian
+ ht, ///< Haitian Creole
+ hu, ///< Hungarian
+ hy, ///< Armenian
+ hz, ///< Herero
+ ia, ///< Interlingua
+ id, ///< Indonesian (formerly in)
+ ie, ///< Interlingue
+ ig, ///< Igbo
+ ii, ///< Nuosu
+ ik, ///< Inupiak
+ io, ///< Ido
+ is, ///< Icelandic
+ it, ///< Italian
+ iu, ///< Inuktitut
+ ja, ///< Japanese
+ jv, ///< Javanese
+ ka, ///< Georgian
+ kg, ///< Kongo
+ ki, ///< Gikuyu
+ kj, ///< Kuanyama
+ kk, ///< Kazakh
+ kl, ///< Greenlandic
+ km, ///< Cambodian
+ kn, ///< Kannada
+ ko, ///< Korean
+ kr, ///< Kanuri
+ ks, ///< Kashmiri
+ ku, ///< Kurdish
+ kv, ///< Komi
+ kw, ///< Cornish
+ ky, ///< Kirghiz
+ la, ///< Latin
+ lb, ///< Letzeburgesch
+ lg, ///< Ganda
+ li, ///< Limburgan; Limburger; Limburgish
+ ln, ///< Lingala
+ lo, ///< Laothian
+ lt, ///< Lithuanian
+ lu, ///< Luba-Katanga
+ lv, ///< Latvian
+ mg, ///< Malagasy
+ mh, ///< Marshallese
+ mi, ///< Maori
+ mk, ///< Macedonian
+ ml, ///< Malayalam
+ mn, ///< Mongolian
+ mo, ///< Moldavian
+ mr, ///< Marathi
+ ms, ///< Malay
+ mt, ///< Maltese
+ my, ///< Burmese
+ na, ///< Nauru
+ nb, ///< Norwegian Bokmal
+ nd, ///< Ndebele, North
+ ne, ///< Nepali
+ ng, ///< Ndonga
+ nl, ///< Dutch
+ nn, ///< Norwegian Nynorsk
+ no, ///< Norwegian
+ nr, ///< Ndebele, South
+ nv, ///< Navajo; Navaho
+ ny, ///< Chichewa; Chewa; Nyanja
+ oc, ///< Occitan
+ oj, ///< Ojibwa
+ om, ///< Oromo
+ or_, ///< Oriya
+ os, ///< Ossetian; Ossetic
+ pa, ///< Panjabi; Punjabi
+ pi, ///< Pali
+ pl, ///< Polish
+ ps, ///< Pashto, Pushto
+ pt, ///< Portuguese
+ qu, ///< Quechua
+ rm, ///< Romansh
+ rn, ///< Kirundi
+ ro, ///< Romanian
+ ru, ///< Russian
+ rw, ///< Kinyarwanda
+ sa, ///< Sanskrit
+ sc, ///< Sardinian
+ sd, ///< Sindhi
+ se, ///< Northern Sami
+ sg, ///< Sangho
+ sh, ///< Serbo-Croatian
+ si, ///< Sinhalese
+ sk, ///< Slovak
+ sl, ///< Slovenian
+ sm, ///< Samoan
+ sn, ///< Shona
+ so, ///< Somali
+ sq, ///< Albanian
+ sr, ///< Serbian
+ ss, ///< Siswati
+ st, ///< Sesotho
+ su, ///< Sundanese
+ sv, ///< Swedish
+ sw, ///< Swahili
+ ta, ///< Tamil
+ te, ///< Telugu
+ tg, ///< Tajik
+ th, ///< Thai
+ ti, ///< Tigrinya
+ tk, ///< Turkmen
+ tl, ///< Tagalog
+ tn, ///< Setswana
+ to, ///< Tonga
+ tr, ///< Turkish
+ ts, ///< Tsonga
+ tt, ///< Tatar
+ tw, ///< Twi
+ ty, ///< Tahitian
+ ug, ///< Uighur
+ uk, ///< Ukrainian
+ ur, ///< Urdu
+ uz, ///< Uzbek
+ ve, ///< Venda
+ vi, ///< Vietnamese
+ vo, ///< Volapuk
+ wa, ///< Walloon
+ wo, ///< Wolof
+ xh, ///< Xhosa
+ yi, ///< Yiddish
+ yo, ///< Yoruba
+ za, ///< Zhuang
+ zh, ///< Chinese
+ zu, ///< Zulu
NUM_ENTRIES
};
}
=== modified file 'include/zorba/stemmer.h'
--- include/zorba/stemmer.h 2012-04-24 12:39:38 +0000
+++ include/zorba/stemmer.h 2012-05-16 22:51:21 +0000
@@ -52,6 +52,23 @@
virtual void destroy() const = 0;
/**
+ * Various properties of this %Stemmer.
+ */
+ struct Properties {
+ /**
+ * The URI that uniquely identifies this %Stemmer.
+ */
+ char const *uri;
+ };
+
+ /**
+ * Gets the Properties of this %Stemmer.
+ *
+ * @param result The Properties to populate.
+ */
+ virtual void properties( Properties *result ) const = 0;
+
+ /**
* Stems the given word.
*
* @param word The word to stem.
@@ -66,7 +83,7 @@
};
/**
- * A %StemmerProvider, given an language, provies a stemmer for it.
+ * A %StemmerProvider, given a language, provides a Stemmer for it.
*/
class ZORBA_DLL_PUBLIC StemmerProvider {
public:
@@ -76,10 +93,12 @@
* Gets a Stemmer for the given language.
*
* @param lang The language to get a Stemmer for.
- * @return The relevant Stemmer or \c NULL if no stemmer for the given
- * language is available.
+ * @param s If not \c null, set to point to a Stemmer for \a lang.
+ * @return Returns \c true only if this provider can provide a stemmer for
+ * \a lang.
*/
- virtual Stemmer::ptr getStemmer( locale::iso639_1::type lang ) const = 0;
+ virtual bool getStemmer( locale::iso639_1::type lang,
+ Stemmer::ptr *s = 0 ) const = 0;
};
///////////////////////////////////////////////////////////////////////////////
=== modified file 'include/zorba/thesaurus.h'
--- include/zorba/thesaurus.h 2012-04-24 12:39:38 +0000
+++ include/zorba/thesaurus.h 2012-05-16 22:51:21 +0000
@@ -32,25 +32,13 @@
///////////////////////////////////////////////////////////////////////////////
/**
- * Contains additional data for URIMappers and URLResolvers
- * when mapping/resolving a Thesaurus URI.
- */
-class ZORBA_DLL_PUBLIC ThesaurusEntityData : public EntityData {
-public:
- /**
- * Gets the language for which a thesaurus is being requested.
- *
- * @return said language.
- */
- virtual locale::iso639_1::type getLanguage() const = 0;
-};
-
-/**
- * A %Thesaurus is-a Resource for thesaurus implementations.
- */
-class ZORBA_DLL_PUBLIC Thesaurus : public Resource {
-public:
- typedef std::unique_ptr<Thesaurus,internal::ztd::destroy_delete<Thesaurus> >
+ * A %Thesaurus provides a way to look up related phrases for a given phrase.
+ */
+class ZORBA_DLL_PUBLIC Thesaurus {
+public:
+ typedef std::unique_ptr<
+ Thesaurus const,internal::ztd::destroy_delete<Thesaurus const>
+ >
ptr;
/**
@@ -88,11 +76,11 @@
* Destroys this %Thesaurus.
* This function is called by Zorba when the %Thesaurus is no longer needed.
*
- * If your URLResolver dynamically allocates %Thesaurus objects, then the
+ * If your implementation dynamically allocates %Thesaurus objects, then your
* implementation can simply be (and usually is) <code>delete this</code>.
*
- * If your URLResolver returns a pointer to a static %Thesaurus object, then
- * the implementation should do nothing.
+ * If your implementation returns a pointer to a static %Thesaurus object,
+ * then your implementation should do nothing.
*/
virtual void destroy() const = 0;
@@ -119,6 +107,32 @@
///////////////////////////////////////////////////////////////////////////////
+/**
+ * A %ThesaurusProvider is-a Resource for providing thesauri for a given
+ * language.
+ */
+class ZORBA_DLL_PUBLIC ThesaurusProvider : public Resource {
+public:
+ typedef std::unique_ptr<
+ ThesaurusProvider const,
+ internal::ztd::destroy_delete<ThesaurusProvider const>
+ >
+ ptr;
+
+ /**
+ * Gets a Thesaurus for the given language.
+ *
+ * @param lang The desired language of the thesaurus.
+ * @param t If not \c null, set to point to a Thesaurus for \a lang.
+ * @return Returns \c true only if this provider can provide a thesaurus for
+ * \a lang.
+ */
+ virtual bool getThesaurus( locale::iso639_1::type lang,
+ Thesaurus::ptr *t = 0 ) const = 0;
+};
+
+///////////////////////////////////////////////////////////////////////////////
+
} // namespace zorba
#endif /* ZORBA_NO_FULL_TEXT */
#endif /* ZORBA_THESAURUS_API_H */
=== modified file 'include/zorba/tokenizer.h'
--- include/zorba/tokenizer.h 2012-04-24 12:39:38 +0000
+++ include/zorba/tokenizer.h 2012-05-16 22:51:21 +0000
@@ -18,6 +18,8 @@
#ifndef ZORBA_TOKENIZER_API_H
#define ZORBA_TOKENIZER_API_H
+#include <vector>
+
#include <zorba/config.h>
#include <zorba/locale.h>
#include <zorba/internal/unique_ptr.h>
@@ -67,8 +69,6 @@
* A %Callback is called once per token.
* This is only internally by Zorba.
* You do not need to derive from this class.
- * The only thing you need to do is call the callback's \c operator() once
- * for each token you parse in \c tokenize().
*/
class Callback {
public:
@@ -77,19 +77,75 @@
virtual ~Callback();
/**
+ * This member-function is called whenever an item that is being tokenized
+ * is entered or exited.
+ *
+ * @param item The item being entered or exited.
+ * @param entering If \c true, the item is being entered; if \c false, the
+ * item is being exited.
+ */
+ virtual void item( Item const &item, bool entering );
+
+ /**
* This member-function is called once per token.
*
* @param utf8_s The UTF-8 token string. It is not null-terminated.
* @param utf8_len The number of bytes in the token string.
+ * @param lang The language of the token.
* @param token_no The token number. Token numbers start at 0.
* @param sent_no The sentence number. Sentence numbers start at 1.
* @param para_no The paragraph number. Paragraph numbers start at 1.
- * @param payload Optional user-defined data.
- */
- virtual void operator()( char const *utf8_s, size_type utf8_len,
- size_type token_no, size_type sent_no,
- size_type para_no, void *payload = 0 ) = 0;
- };
+ * @param item The Item this token is from, if any.
+ */
+ virtual void token( char const *utf8_s, size_type utf8_len,
+ locale::iso639_1::type lang,
+ size_type token_no, size_type sent_no,
+ size_type para_no, Item const *item = 0 ) = 0;
+ };
+
+ /////////////////////////////////////////////////////////////////////////////
+
+ /**
+ * Various properties of this %Tokenizer.
+ */
+ struct Properties {
+ typedef std::vector<locale::iso639_1::type> languages_type;
+
+ /**
+ * If \c true, XML comments separate tokens. For example,
+ * \c net<!---->work would be 2 tokens instead of 1.
+ */
+ bool comments_separate_tokens;
+
+ /**
+ * If \c true, XML elements separate tokens. For example,
+ * \c <b>B</b>old would be 2 tokens instead of 1.
+ */
+ bool elements_separate_tokens;
+
+ /**
+ * If \c true, XML processing instructions separate tokens. For example,
+ * <code>net<?PI pi?>work</code> would be 2 tokens instead of 1.
+ */
+ bool processing_instructions_separate_tokens;
+
+ /**
+ * The set of languages supported.
+ */
+ languages_type languages;
+
+ /**
+ * The URI that uniquely identifies this %Tokenizer.
+ */
+ char const* uri;
+ };
+
+ /**
+ * Gets the Properties of this %Tokenizer.
+ *
+ * @param result The Properties to populate.
+ */
+ virtual void properties( Properties *result ) const = 0;
/////////////////////////////////////////////////////////////////////////////
@@ -106,39 +162,6 @@
virtual void destroy() const = 0;
/**
- * Trace options for XML elements combined via bitwise-or.
- */
- enum ElementTraceOptions {
- trace_none = 0x0, ///< Trace no elements.
- trace_begin = 0x1, ///< Trace the beginning of elements.
- trace_end = 0x2 ///< Trace the ending of elements.
- };
-
- /**
- * Gets the trace options. If the value is \c trace_none, then the paragraph
- * number will be incremented upon entering an XML element; if the value is
- * anything other than \c trace_none, then the tokenizer assumes
- * responsibility for incrementing the paragraph number.
- *
- * @return Returns said options.
- */
- int trace_options() const {
- return trace_options_;
- }
-
- /**
- * This function is called whenever an XML element is entered during
- * tokenization. Note that this function is called only if \c
- * trace_options() returns non-zero.
- *
- * @param qname The element's QName.
- * @param trace_options The bitwise-or of the trace option(s) in effect for a
- * particular call.
- * @see trace_options()
- */
- virtual void element( Item const &qname, int trace_options );
-
- /**
* Gets this %Tokenizer's associated Numbers.
*
* @return Returns said Numbers.
@@ -153,6 +176,16 @@
Numbers const& numbers() const;
/**
+ * Tokenizes the given node.
+ *
+ * @param node The node to tokenize.
+ * @param lang The default language to use.
+ * @param callback The Callback to call once per token.
+ */
+ void tokenize_node( Item const &node, locale::iso639_1::type lang,
+ Callback &callback );
+
+ /**
* Tokenizes the given string.
*
* @param utf8_s The UTF-8 string to tokenize. It need not be
@@ -162,11 +195,11 @@
* @param wildcards If \c true, allows XQuery wildcard syntax characters to
* be part of tokens.
* @param callback The Callback to call once per token.
- * @param payload Optional user-defined data.
+ * @param item The Item this string is from, if any.
*/
- virtual void tokenize( char const *utf8_s, size_type utf8_len,
- locale::iso639_1::type lang, bool wildcards,
- Callback &callback, void *payload = 0 ) = 0;
+ virtual void tokenize_string( char const *utf8_s, size_type utf8_len,
+ locale::iso639_1::type lang, bool wildcards,
+ Callback &callback, Item const *item = 0 ) = 0;
/////////////////////////////////////////////////////////////////////////////
@@ -175,27 +208,71 @@
* Constructs a %Tokenizer.
*
* @param numbers the Numbers to use.
- * @param trace_options The bitwise-or of the available trace options, if
- * any.
*/
- Tokenizer( Numbers &numbers, int trace_options = trace_none );
+ Tokenizer( Numbers &numbers );
/**
* Destroys a %Tokenizer.
*/
virtual ~Tokenizer() = 0;
+ /**
+ * Given an element, finds its \c xml:lang attribute, if any, and gets its
+ * value.
+ *
+ * @param element The element to check.
+ * @param lang A pointer to where to put the found language, if any.
+ * @return Returns \c true only if an \c xml:lang attribute is found and the
+ * value is a known language.
+ */
+ bool find_lang_attribute( Item const &element, locale::iso639_1::type *lang );
+
+ /**
+ * This member-function is called whenever an item that is being tokenized is
+ * entered or exited.
+ *
+ * @param item The item being entered or exited.
+ * @param entering If \c true, the item is being entered; if \c false, the
+ * item is being exited.
+ */
+ virtual void item( Item const &item, bool entering );
+
+ /**
+ * Tokenizes the given node and all of its child nodes, if any. For each
+ * node, it is required that this function call the item() member function of
+ * both this %Tokenizer and of the Callback twice, once each for entrance and
+ * exit.
+ *
+ * @param node The node to tokenize.
+ * @param lang The default language to use.
+ * @param callback The Callback to call per token.
+ * @param tokenize_acp If \c true, additionally tokenize all attribute,
+ * comment, and processing-instruction nodes encountered;
+ * if \c false, skip them.
+ */
+ virtual void tokenize_node_impl( Item const &node,
+ locale::iso639_1::type lang,
+ Callback &callback, bool tokenize_acp );
+
private:
- int trace_options_;
- Numbers *no_;
+ Numbers *numbers_;
};
+inline Tokenizer::Tokenizer( Numbers &numbers ) : numbers_( &numbers ) {
+}
+
inline Tokenizer::Numbers& Tokenizer::numbers() {
- return *no_;
+ return *numbers_;
}
inline Tokenizer::Numbers const& Tokenizer::numbers() const {
- return *no_;
+ return *numbers_;
+}
+
+inline void Tokenizer::tokenize_node( Item const &item,
+ locale::iso639_1::type lang,
+ Callback &callback ) {
+ tokenize_node_impl( item, lang, callback, true );
}
///////////////////////////////////////////////////////////////////////////////
@@ -211,11 +288,14 @@
* Creates a new %Tokenizer.
*
* @param lang The language of the text that the tokenizer will tokenize.
- * @param numbers The Numbers to use.
- * @return Returns said %Tokenizer.
+ * @param numbers The Numbers to use. If \c null, \a t is not set.
+ * @param t If not \c null, set to point to a Tokenizer for \a lang.
+ * @return Returns \c true only if this provider can provide a tokenizer for
+ * \a lang.
*/
- virtual Tokenizer::ptr getTokenizer( locale::iso639_1::type lang,
- Tokenizer::Numbers &numbers ) const = 0;
+ virtual bool getTokenizer( locale::iso639_1::type lang,
+ Tokenizer::Numbers *numbers = 0,
+ Tokenizer::ptr *t = 0 ) const = 0;
};
///////////////////////////////////////////////////////////////////////////////
=== modified file 'include/zorba/transcode_stream.h'
--- include/zorba/transcode_stream.h 2012-04-24 12:39:38 +0000
+++ include/zorba/transcode_stream.h 2012-05-16 22:51:21 +0000
@@ -58,6 +58,7 @@
* os.ios::rdbuf( tbuf.orig_streambuf() );
* throw;
* }
+ * os.ios::rdbuf( tbuf.orig_streambuf() );
* }
* \endcode
* Alternatively, you may wish to use either \c attach(), \c auto_attach, or
@@ -254,7 +255,14 @@
* @throws std::invalid_argument if \a charset is not supported.
*/
stream( char const *charset ) :
+#ifdef WIN32
+# pragma warning( push )
+# pragma warning( disable : 4355 )
+#endif /* WIN32 */
tbuf_( charset, this->rdbuf() )
+#ifdef WIN32
+# pragma warning( pop )
+#endif /* WIN32 */
{
init();
}
@@ -272,7 +280,14 @@
template<typename StreamArgType>
stream( char const *charset, StreamArgType stream_arg ) :
StreamType( stream_arg ),
+#ifdef WIN32
+# pragma warning( push )
+# pragma warning( disable : 4355 )
+#endif /* WIN32 */
tbuf_( charset, this->rdbuf() )
+#ifdef WIN32
+# pragma warning( pop )
+#endif /* WIN32 */
{
init();
}
@@ -292,7 +307,14 @@
stream( char const *charset, StreamArgType stream_arg,
std::ios_base::openmode mode ) :
StreamType( stream_arg, mode ),
+#ifdef WIN32
+# pragma warning( push )
+# pragma warning( disable : 4355 )
+#endif /* WIN32 */
tbuf_( charset, this->rdbuf() )
+#ifdef WIN32
+# pragma warning( pop )
+#endif /* WIN32 */
{
init();
}
=== modified file 'include/zorba/uri_resolvers.h'
--- include/zorba/uri_resolvers.h 2012-04-24 12:39:38 +0000
+++ include/zorba/uri_resolvers.h 2012-05-16 22:51:21 +0000
@@ -50,7 +50,8 @@
class ZORBA_DLL_PUBLIC Resource
{
public:
- typedef std::unique_ptr<Resource,internal::ztd::destroy_delete<Resource> > ptr;
+ typedef std::unique_ptr<Resource,internal::ztd::destroy_delete<Resource> >
+ ptr;
virtual ~Resource() = 0;
@@ -172,8 +173,8 @@
* object itself will be discarded.
*
* In any case, if they create a Resource, Zorba will take memory
- * ownership of the Resource and delete it when it is no longer
- * needed.
+ * ownership of the Resource and delete it (by calling destroy() on it)
+ * when it is no longer needed.
*/
virtual Resource* resolveURL(const zorba::String& aUrl,
EntityData const* aEntityData) = 0;
@@ -268,7 +269,6 @@
* Constructor. Specify the Entity Kind you wish to map. Optionally,
* specify whether this should be a CANDIDATE or COMPONENT mapper;
* default is CANDIDATE.
- * QQQ COMPONENT is no longer used; delete?
*/
OneToOneURIMapper(EntityData::Kind aEntityKind,
URIMapper::Kind aMapperKind = URIMapper::CANDIDATE);
=== modified file 'modules/ExternalModules.conf'
--- modules/ExternalModules.conf 2012-04-26 20:54:34 +0000
+++ modules/ExternalModules.conf 2012-05-16 22:51:21 +0000
@@ -26,7 +26,7 @@
# "tag" is the VCS tag to check out (optional - defaults to HEAD;
# currently only works for bzr, since svn tags are just different URLS)
-data-cleaning bzr lp:zorba/data-cleaning-module zorba-2.2
+data-cleaning bzr lp:zorba/data-cleaning-module 1.1
data-converters bzr lp:zorba/data-converters-module zorba-2.2
data-formatting bzr lp:zorba/data-formatting-module zorba-2.2-jvm
excel bzr lp:zorba/excel-module zorba-2.2
@@ -34,7 +34,7 @@
http-client bzr lp:zorba/http-client-module zorba-2.2
image bzr lp:zorba/image-module 1.0
languages bzr lp:zorba/languages-module zorba-2.2
-oauth bzr lp:zorba/oauth-module zorba-2.2
+oauth bzr lp:zorba/oauth-module 1.0
process bzr lp:zorba/process-module zorba-2.2
security bzr lp:zorba/security-module zorba-2.2
system bzr lp:zorba/system-module zorba-2.2
@@ -42,3 +42,5 @@
email bzr lp:zorba/email-module zorba-2.2
util-jvm bzr lp:zorba/util-jvm-module
schema-tools bzr lp:zorba/schema-tools-module
+stack bzr lp:zorba/stack-module
+queue bzr lp:zorba/queue-module
=== modified file 'modules/com/zorba-xquery/www/modules/CMakeLists.txt'
--- modules/com/zorba-xquery/www/modules/CMakeLists.txt 2012-04-24 12:39:38 +0000
+++ modules/com/zorba-xquery/www/modules/CMakeLists.txt 2012-05-16 22:51:21 +0000
@@ -72,6 +72,13 @@
DECLARE_ZORBA_MODULE(FILE xqdoc.xq VERSION 2.0
URI "http://www.zorba-xquery.com/modules/xqdoc";)
+IF(NOT ZORBA_NO_FULL_TEXT)
+ DECLARE_ZORBA_MODULE(FILE full-text.xq VERSION 2.0
+ URI "http://www.zorba-xquery.com/modules/full-text";)
+ DECLARE_ZORBA_SCHEMA(FILE full-text.xsd
+ URI "http://www.zorba-xquery.com/modules/full-text";)
+ENDIF(NOT ZORBA_NO_FULL_TEXT)
+
# Subdirectories
DECLARE_ZORBA_MODULE(FILE converters/base64.xq VERSION 2.0
URI "http://www.zorba-xquery.com/modules/converters/base64";)
=== modified file 'modules/com/zorba-xquery/www/modules/debugger/dbgp-message-handler.xq'
--- modules/com/zorba-xquery/www/modules/debugger/dbgp-message-handler.xq 2012-04-24 12:39:38 +0000
+++ modules/com/zorba-xquery/www/modules/debugger/dbgp-message-handler.xq 2012-05-16 22:51:21 +0000
@@ -258,7 +258,7 @@
: @param $message the message.
: @return ().
:)
-declare function dmh:process($message as element())
+declare function dmh:process($message as element()) as xs:anyAtomicType*
{
let $nodeName := fn:local-name($message)
let $id := fn:data($message/@transaction_id)
=== added file 'modules/com/zorba-xquery/www/modules/full-text.xq'
--- modules/com/zorba-xquery/www/modules/full-text.xq 1970-01-01 00:00:00 +0000
+++ modules/com/zorba-xquery/www/modules/full-text.xq 2012-05-16 22:51:21 +0000
@@ -0,0 +1,839 @@
+xquery version "3.0";
+
+(:
+ : Copyright 2006-2011 The FLWOR Foundation.
+ :
+ : Licensed under the Apache License, Version 2.0 (the "License");
+ : you may not use this file except in compliance with the License.
+ : You may obtain a copy of the License at
+ :
+ : http://www.apache.org/licenses/LICENSE-2.0
+ :
+ : Unless required by applicable law or agreed to in writing, software
+ : distributed under the License is distributed on an "AS IS" BASIS,
+ : WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ : See the License for the specific language governing permissions and
+ : limitations under the License.
+ :)
+
+(:===========================================================================:)
+
+(:~
+ : This module provides an XQuery API to full-text functions.
+ : For general information about Zorba's implementation of the
+ : <a href="http://www.w3.org/TR/xpath-full-text-10/";>XQuery and XPath Full Text 1.0 specification</a>
+ : as well as instructions for building an installing a thesaurus,
+ : see the <a href="http://www.zorba-xquery.com/html/documentation/latest/zorba/ft_thesaurus";>Full Text Thesaurus documentation</a>.
+ : <h2>Notes on languages</h2>
+ : To refer to paricular human languages,
+ : Zorba uses both the
+ : <a href="http://en.wikipedia.org/wiki/ISO_639-1";>ISO 639-1</a>
+ : and
+ : <a href="http://en.wikipedia.org/wiki/ISO_639-2";>ISO 639-2</a>
+ : languages codes.
+ : Note that Zorba supports only a subset of the
+ : <a href="http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes";>complete list of language codes</a>
+ : and not every function supports the same subset.
+ : <p/>
+ : Most functions in this module take a language as a parameter
+ : using the
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>
+ : XML schema data type.
+ : <h2>Notes on stemming</h2>
+ : The <code>stem()</code> functions return the
+ : <a href="http://en.wikipedia.org/wiki/Word_stem";>stem</a>
+ : of a word.
+ : In Zorba,
+ : the stem of a word itself, however, is not guaranteed to be a word.
+ : It is best to consider a stem as an opaque byte sequence.
+ : All that is guaranteed about a stem is that,
+ : for a given word,
+ : the stem of that word will always be the same byte sequence.
+ : Hence,
+ : you sould never compare the result of one of the <code>stem()</code>
+ : functions against a non-stemmed string,
+ : for example:
+ : <pre>
+ : if ( ft:stem( "apples" ) eq "apple" ) ** WRONG **
+ : </pre>
+ : Instead do:
+ : <pre>
+ : if ( ft:stem( "apples" ) eq ft:stem( "apple" ) ) ** CORRECT **
+ : </pre>
+ : <h2>Notes on the thesaurus</h2>
+ : The <code>thesaurus-lookup()</code> functions have "levels"
+ : and "relationship" parameters.
+ : The values for these are implementation-defined.
+ : Zorba's default implementation uses the
+ : <a href="http://wordnet.princeton.edu/";>WordNet lexical database</a>,
+ : version 3.0.
+ : <p/>
+ : In WordNet,
+ : the number of "levels" that two phrases are apart
+ : are how many hierarchical meanings apart they are.
+ : For example,
+ : "canary" is 5 levels away from "vertebrate"
+ : (carary > finch > oscine > passerine > bird > vertebrate).
+ : <p/>
+ : When using the WordNet implementation,
+ : Zorba supports all of the relationships (and their abbreviations)
+ : specified by
+ : <a href="http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=7776";>ISO 2788</a>
+ : and
+ : <a href="http://www.niso.org/kst/reports/standards?step=2&gid=&project_key=7cc9b583cb5a62e8c15d3099e0bb46bbae9cf38a";>ANSI/NISO Z39.19-2005</a>
+ : with the exceptions of "HN" (history note)
+ : and "X SN" (see scope note for).
+ : These relationships are:
+ : <table>
+ : <tr>
+ : <th>Rel.</th>
+ : <th>Meaning</th>
+ : <th>WordNet Rel.</th>
+ : </tr>
+ : <tr>
+ : <td>BT</td>
+ : <td>broader term</td>
+ : <td>hypernym</td>
+ : </tr>
+ : <tr>
+ : <td>BTG</td>
+ : <td>broader term generic</td>
+ : <td>hypernym</td>
+ : </tr>
+ : <tr>
+ : <td>BTI</td>
+ : <td>broader term instance</td>
+ : <td>instance hypernym</td>
+ : </tr>
+ : <tr>
+ : <td>BTP</td>
+ : <td>broader term partitive</td>
+ : <td>part meronym</td>
+ : </tr>
+ : <tr>
+ : <td>NT</td>
+ : <td>narrower term</td>
+ : <td>hyponym</td>
+ : </tr>
+ : <tr>
+ : <td>NTG</td>
+ : <td>narrower term generic</td>
+ : <td>hyponym</td>
+ : </tr>
+ : <tr>
+ : <td>NTI</td>
+ : <td>narrower term instance</td>
+ : <td>instance hyponym</td>
+ : </tr>
+ : <tr>
+ : <td>NTP</td>
+ : <td>narrower term partitive</td>
+ : <td>part holonym</td>
+ : </tr>
+ : <tr>
+ : <td>RT</td>
+ : <td>related term</td>
+ : <td>also see</td>
+ : </tr>
+ : <tr>
+ : <td>SN</td>
+ : <td>scope note</td>
+ : <td>n/a</td>
+ : </tr>
+ : <tr>
+ : <td>TT</td>
+ : <td>top term</td>
+ : <td>hypernym</td>
+ : </tr>
+ : <tr>
+ : <td>UF</td>
+ : <td>non-preferred term</td>
+ : <td>n/a</td>
+ : </tr>
+ : <tr>
+ : <td>USE</td>
+ : <td>preferred term</td>
+ : <td>n/a</td>
+ : </tr>
+ : </table>
+ : Note that you can specify relationships
+ : either by their abbreviation
+ : or their meaning.
+ : Relationships are case-insensitive.
+ :
+ : In addition to the
+ : <a href="http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=7776";>ISO 2788</a>
+ : and
+ : <a href="http://www.niso.org/kst/reports/standards?step=2&gid=&project_key=7cc9b583cb5a62e8c15d3099e0bb46bbae9cf38a";>ANSI/NISO Z39.19-2005</a>
+ : relationships,
+ : Zorba also supports all of the relationships offered by WordNet.
+ : These relationships are:
+ : <table class="ft_rels">
+ : <tr>
+ : <th>Relationship</th>
+ : <th>Meaning</th>
+ : </tr>
+ : <tr>
+ : <td nowrap="nowrap">also see</td>
+ : <td>
+ : A word that is related to another,
+ : e.g., for "varnished" (furniture)
+ : one should <em>also see</em> "finished."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td>antonym</td>
+ : <td>
+ : A word opposite in meaning to another,
+ : e.g., "light" is an <em>antonym</em> for "heavy."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td>attribute</td>
+ : <td>
+ : A noun for which adjectives express values,
+ : e.g., "weight" is an <em>attribute</em>
+ : for which the adjectives "light" and "heavy"
+ : express values.
+ : </td>
+ : </tr>
+ : <tr>
+ : <td>cause</td>
+ : <td>
+ : A verb that causes another,
+ : e.g., "show" is a <em>cause</em> of "see."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td nowrap="nowrap">derivationally related form</td>
+ : <td>
+ : A word that is derived from a root word,
+ : e.g., "metric" is a <em>derivationally related form</em> of "meter."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td nowrap="nowrap">derived from adjective</td>
+ : <td>
+ : An adverb that is derived from an adjective,
+ : e.g., "correctly" is <em>derived from the adjective</em> "correct."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td>entailment</td>
+ : <td>
+ : A verb that presupposes another,
+ : e.g., "snoring" <em>entails</em> "sleeping."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td>hypernym</td>
+ : <td>
+ : A word with a broad meaning that more specific words fall under,
+ : e.g., "meal" is a <em>hypernym</em> of "breakfast."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td>hyponym</td>
+ : <td>
+ : A word of more specific meaning than a general term applicable to it,
+ : e.g., "breakfast" is a <em>hyponym</em> of "meal."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td nowrap="nowrap">instance hypernym</td>
+ : <td>
+ : A word that denotes a category of some specific instance,
+ : e.g., "author" is an <em>instance hypernym</em> of "Asimov."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td nowrap="nowrap">instance hyponym</td>
+ : <td>
+ : A term that donotes a specific instance of some general category,
+ : e.g., "Asimov" is an <em>instance hyponym</em> of "author."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td nowrap="nowrap">member holonym</td>
+ : <td>
+ : A word that denotes a collection of individuals,
+ : e.g., "faculty" is a <em>member holonym</em> of "professor."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td nowrap="nowrap">member meronym</td>
+ : <td>
+ : A word that denotes a member of a larger group,
+ : e.g., a "person" is a <em>member meronym</em> of a "crowd."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td nowrap="nowrap">part holonym</td>
+ : <td>
+ : A word that denotes a larger whole comprised of some part,
+ : e.g., "car" is a <em>part holonym</em> of "engine."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td nowrap="nowrap">part meronym</td>
+ : <td>
+ : A word that denotes a part of a larger whole,
+ : e.g., an "engine" is <em>part meronym</em> of a "car."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td nowrap="nowrap">participle of verb</td>
+ : <td>
+ : An adjective that is the participle of some verb,
+ : e.g., "breaking" is the <em>participle of the verb</em> "break."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td>pertainym</td>
+ : <td>
+ : An adjective that classifies its noun,
+ : e.g., "musical" is a <em>pertainym</em> in "musical instrument."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td nowrap="nowrap">similar to</td>
+ : <td>
+ : Similar, though not necessarily interchangeable, adjectives.
+ : For example, "shiny" is <em>similar to</em> "bright",
+ : but they have subtle differences.
+ : </td>
+ : </tr>
+ : <tr>
+ : <td nowrap="nowrap">substance holonym</td>
+ : <td>
+ : A word that denotes a larger whole containing some constituent
+ : substance, e.g., "bread" is a <em>substance holonym</em> of "flour."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td nowrap="nowrap">substance meronym</td>
+ : <td>
+ : A word that denotes a constituant substance of some larger whole,
+ : e.g., "flour" is a <em>substance meronym</em> of "bread."
+ : </td>
+ : </tr>
+ : <tr>
+ : <td nowrap="nowrap">verb group</td>
+ : <td>
+ : A verb that is a member of a group of similar verbs,
+ : e.g., "live" is in the <em>verb group</em>
+ : of "dwell", "live", "inhabit", etc.
+ : </td>
+ : </tr>
+ : </table>
+ : <h2>Notes on tokenization</h2>
+ : For general information about Zorba's implementation of tokenization,
+ : including what constitutes a token,
+ : see the <a href="http://www.zorba-xquery.com/html/documentation/latest/zorba/ft_tokenizer";>Full Text Tokenizer</a> documentation.
+ : @author Paul J. Lucas
+ :)
+module namespace ft = "http://www.zorba-xquery.com/modules/full-text";;
+
+import schema namespace ft-schema =
+ "http://www.zorba-xquery.com/modules/full-text";;
+
+declare namespace err = "http://www.w3.org/2005/xqt-errors";;
+declare namespace zerr = "http://www.zorba-xquery.com/errors";;
+
+declare namespace ver = "http://www.zorba-xquery.com/options/versioning";;
+declare option ver:module-version "2.0";
+
+(:===========================================================================:)
+
+(:~
+ : Predeclared constant for the Danish
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-da as xs:language := xs:language("da");
+
+(:~
+ : Predeclared constant for the German
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-de as xs:language := xs:language("de");
+
+(:~
+ : Predeclared constant for the English
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-en as xs:language := xs:language("en");
+
+(:~
+ : Predeclared constant for the Spanish
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-es as xs:language := xs:language("es");
+
+(:~
+ : Predeclared constant for the Finnish
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-fi as xs:language := xs:language("fi");
+
+(:~
+ : Predeclared constant for the French
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-fr as xs:language := xs:language("fr");
+
+(:~
+ : Predeclared constant for the Hungarian
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-hu as xs:language := xs:language("hu");
+
+(:~
+ : Predeclared constant for the Italian
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-it as xs:language := xs:language("it");
+
+(:~
+ : Predeclared constant for the Dutch
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-nl as xs:language := xs:language("nl");
+
+(:~
+ : Predeclared constant for the Norwegian
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-no as xs:language := xs:language("no");
+
+(:~
+ : Predeclared constant for the Portuguese
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-pt as xs:language := xs:language("pt");
+
+(:~
+ : Predeclared constant for the Romanian
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-ro as xs:language := xs:language("ro");
+
+(:~
+ : Predeclared constant for the Russian
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-ru as xs:language := xs:language("ru");
+
+(:~
+ : Predeclared constant for the Swedish
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-sv as xs:language := xs:language("sv");
+
+(:~
+ : Predeclared constant for the Turkish
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-tr as xs:language := xs:language("tr");
+
+(:===========================================================================:)
+
+(:~
+ : Gets the current
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>:
+ : either the language specified by the
+ : <code><a href="http://www.w3.org/TR/xpath-full-text-10/#doc-xquery10-FTOptionDecl";>declare ft-option using</a>
+ : <a href="http://www.w3.org/TR/xpath-full-text-10/#ftlanguageoption";>language</a></code>
+ : statement (if any)
+ : or the one returned by <code>ft:host-lang()</code> (if none).
+ :
+ : @return said language.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-current-lang-true-1.xq
+ :)
+declare function ft:current-lang()
+ as xs:language external;
+
+(:~
+ : Gets the host's current
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>.
+ : The "host" is the computer on which Zorba is running.
+ : The host's current language is obtained as follows:
+ : <ul>
+ : <li>
+ : For *nix systems:
+ : <ol>
+ : <li>
+ : If <a ref="http://www.cplusplus.com/reference/clibrary/clocale/setlocale/";><code>setlocale</code>(3)</a> returns non-null,
+ : the language corresponding to that locale is used.
+ : </li>
+ : <li>
+ : Else, if the <code>LANG</code> environment variable is set,
+ : that language is ued.
+ : </li>
+ : <li>
+ : Otherwise, there is no default language.
+ : </li>
+ : </ol>
+ : </li>
+ : <li>
+ : For Windows systems,
+ : the language corresponding to the locale returned by the
+ : <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/dd318101(v=vs.85).aspx"><code>GetLocaleInfo()</code></a>
+ : function is used.
+ : </li>
+ : </ul>
+ :
+ : @return said language.
+ :)
+declare function ft:host-lang()
+ as xs:language external;
+
+(:~
+ : Checks whether the given
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : is supported for stemming.
+ :
+ : @param $lang The language to check.
+ : @return <code>true</code> only if the language is supported.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stem-lang-es-supported-true.xq
+ :)
+declare function ft:is-stem-lang-supported( $lang as xs:language )
+ as xs:boolean external;
+
+(:~
+ : Checks whether the given word is a stop-word.
+ :
+ : @param $word The word to check.
+ : @param $lang The
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : of <code>$word</code>.
+ : @return <code>true</code> only if <code>$word</code> is a stop-word.
+ : @error err:FTST0009 if <code>$lang</code> is not supported.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-1.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-3.xq
+ :)
+declare function ft:is-stop-word( $word as xs:string, $lang as xs:language )
+ as xs:boolean external;
+
+(:~
+ : Checks whether the given word is a stop-word.
+ :
+ : @param $word The word to check.
+ : The word's <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : is assumed to be the one returned by <code>ft:current-lang()</code>.
+ : @return <code>true</code> only if <code>$word</code> is a stop-word.
+ : @error err:FTST0009 if <code>ft:current-lang()</code> is not supported.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-2.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-4.xq
+ :)
+declare function ft:is-stop-word( $word as xs:string )
+ as xs:boolean external;
+
+(:~
+ : Checks whether the given
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : is supported for stop words.
+ :
+ : @param $lang The language to check.
+ : @return <code>true</code> only if the language is supported.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-en-supported-true.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-supported-false-1.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-supported-false-2.xq
+ :)
+declare function ft:is-stop-word-lang-supported( $lang as xs:language )
+ as xs:boolean external;
+
+(:~
+ : Checks whether the given
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : is supported for look-up using the default thesaurus.
+ :
+ : @param $lang The language to check.
+ : @return <code>true</code> only if the language is supported.
+ :)
+declare function ft:is-thesaurus-lang-supported( $lang as xs:language )
+ as xs:boolean external;
+
+(:~
+ : Checks whether the given
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : is supported for look-up using the thesaurus specified by the given URI.
+ :
+ : @param $uri The URI specifying the thesaurus to use.
+ : @param $lang The language to check.
+ : @return <code>true</code> only if the language is supported.
+ : @error err:FTST0018 if <code>$uri</code> refers to a thesaurus
+ : that is not found in the statically known thesauri.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-thesaurus-lang-supported-true-1.xq
+ :)
+declare function ft:is-thesaurus-lang-supported( $uri as xs:string,
+ $lang as xs:language )
+ as xs:boolean external;
+
+(:~
+ : Checks whether the given
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : is supported for tokenization.
+ :
+ : @param $lang The language to check.
+ : @return <code>true</code> only if the language is supported.
+ :)
+declare function ft:is-tokenizer-lang-supported( $lang as xs:language )
+ as xs:boolean external;
+
+(:~
+ : Stems the given word.
+ :
+ : @param $word The word to stem.
+ : @param $lang The
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : of <code>$word</code>.
+ : @return the stem of <code>$word</code>.
+ : @error err:FTST0009 if <code>$lang</code> is not supported.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-1.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-2.xq
+ :)
+declare function ft:stem( $word as xs:string, $lang as xs:language )
+ as xs:string external;
+
+(:~
+ : Stems the given word.
+ :
+ : @param $word The word to stem.
+ : The word's <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : is assumed to be the one returned by <code>ft:current-lang()</code>.
+ : @return the stem of <code>$word</code>.
+ : @error err:FTST0009 if <code>ft:current-lang()</code> is not supported.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-3.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-4.xq
+ :)
+declare function ft:stem( $word as xs:string )
+ as xs:string external;
+
+(:~
+ : Strips all diacritical marks from all characters.
+ :
+ : @param $string The string to strip diacritical marks from.
+ : @return <code>$string</code> with diacritical marks stripped.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-strip-diacritics-1.xq
+ :)
+declare function ft:strip-diacritics( $string as xs:string )
+ as xs:string external;
+
+(:~
+ : Looks-up the given phrase in the default thesaurus.
+ :
+ : @param $phrase The phrase to look up.
+ : The phrase's
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : is assumed to be the one returned by <code>ft:current-lang()</code>.
+ : @return the original and related phrases.
+ : @error err:FTST0009 if <code>ft:current-lang()</code> is not supported.
+ : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
+ : by the currently running version of Zorba.
+ : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
+ : that of the CPU on which Zorba is currently running.
+ : @error zerr:ZXQP8403 if there was an error reading the thesaurus data.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-1.xq
+ :)
+declare function ft:thesaurus-lookup( $phrase as xs:string )
+ as xs:string+ external;
+
+(:~
+ : Looks-up the given phrase in the thesaurus specified by the given URI.
+ :
+ : @param $uri The URI specifying the thesaurus to use.
+ : @param $phrase The phrase to look up.
+ : @param $lang The
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : of <code>$phrase</code>.
+ : @return the original and related phrases.
+ : @error err:FTST0009 if <code>$lang</code> is not supported.
+ : @error err:FTST0018 if <code>$uri</code> refers to a thesaurus
+ : that is not found in the statically known thesauri.
+ : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
+ : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
+ : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
+ : by the currently running version of Zorba.
+ : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
+ : that of the CPU on which Zorba is currently running.
+ : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-2.xq
+ :)
+declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
+ $lang as xs:language )
+ as xs:string+ external;
+
+(:~
+ : Looks-up the given phrase in a thesaurus.
+ :
+ : @param $uri The URI specifying the thesaurus to use.
+ : @param $phrase The phrase to look up.
+ : The phrase's
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : is assumed to be the one the one returned by <code>ft:current-lang()</code>.
+ : @return the original and related phrases.
+ : @error err:FTST0009 if <code>ft:current-lang()</code> is unsupported.
+ : @error err:FTST0018 if <code>$uri</code> refers to a thesaurus
+ : that is not found in the statically known thesauri.
+ : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
+ : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
+ : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
+ : by the currently running version of Zorba.
+ : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
+ : that of the CPU on which Zorba is currently running.
+ : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-3.xq
+ :)
+declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string )
+ as xs:string+ external;
+
+(:~
+ : Looks-up the given phrase in a thesaurus.
+ :
+ : @param $uri The URI specifying the thesaurus to use.
+ : @param $phrase The phrase to look up.
+ : @param $lang The
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : of <code>$phrase</code>.
+ : @param $relationship The relationship the results are to have to
+ : <code>$phrase</code>.
+ : @return the original and related phrases.
+ : @error err:FTST0018 if <code>$uri</code> refers to a thesaurus
+ : that is not found in the statically known thesauri.
+ : @error err:FTST0009 if <code>$lang</code> is not supported.
+ : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
+ : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
+ : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
+ : by the currently running version of Zorba.
+ : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
+ : that of the CPU on which Zorba is currently running.
+ : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-4.xq
+ :)
+declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
+ $lang as xs:language,
+ $relationship as xs:string )
+ as xs:string+ external;
+
+(:~
+ : Looks-up the given phrase in a thesaurus.
+ :
+ : @param $uri The URI specifying the thesaurus to use.
+ : @param $phrase The phrase to look up.
+ : @param $lang The
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : of <code>$phrase</code>.
+ : @param $relationship The relationship the results are to have to
+ : <code>$phrase</code>.
+ : @param $level-least The minimum number of levels within the thesaurus to be
+ : travers$ed.
+ : @param $level-most The maximum number of levels within the thesaurus to be
+ : traversed.
+ : @return the original and related phrases.
+ : @error err:FOCA0003 if either <code>$level-least</code> or
+ : <code>$level-most</code> is either negative or too large.
+ : @error err:FTST0018 if <code>$uri</code> refers to a thesaurus
+ : that is not found in the statically known thesauri.
+ : @error err:FTST0009 if <code>$lang</code> is not supported.
+ : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
+ : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
+ : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
+ : by the currently running version of Zorba.
+ : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
+ : that of the CPU on which Zorba is currently running.
+ : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-5.xq
+ :)
+declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
+ $lang as xs:language,
+ $relationship as xs:string,
+ $level-least as xs:integer,
+ $level-most as xs:integer )
+ as xs:string+ external;
+
+(:~
+ : Tokenizes the given document.
+ :
+ : @param $node The node to tokenize.
+ : @param $lang The default
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : of <code>$node</code>.
+ : @return a (possibly empty) sequence of tokens.
+ : @error err:FTST0009 if <code>$lang</code> is not supported in general.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-1.xq
+ :)
+declare function ft:tokenize( $node as node(), $lang as xs:language )
+ as element(ft-schema:token)* external;
+
+(:~
+ : Tokenizes the given document.
+ :
+ : @param $node The node to tokenize.
+ : The document's default
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : is assumed to be the one returned by <code>ft:current-lang()</code>.
+ : @return a (possibly empty) sequence of tokens.
+ : @error err:FTST0009 if <code>ft:current-lang()</code> is not supported in
+ : general.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-2.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-3.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-4.xq
+ :)
+declare function ft:tokenize( $node as node() )
+ as element(ft-schema:token)* external;
+
+(:~
+ : Tokenizes the given string.
+ :
+ : @param $string The string to tokenize.
+ : @param $lang The default
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : of <code>$string</code>.
+ : @return a (possibly empty) sequence of tokens.
+ : @error err:FTST0009 if <code>$lang</code> is not supported.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-string-1.xq
+ :)
+declare function ft:tokenize-string( $string as xs:string,
+ $lang as xs:language )
+ as xs:string* external;
+
+(:~
+ : Tokenizes the given string.
+ :
+ : @param $string The string to tokenize.
+ : The string's default
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : is assumed to be the one returned by <code>ft:current-lang()</code>.
+ : @return a (possibly empty) sequence of tokens.
+ : @error err:FTST0009 if <code>ft:current-lang()</code> is not supported.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-string-2.xq
+ :)
+declare function ft:tokenize-string( $string as xs:string )
+ as xs:string* external;
+
+(:~
+ : Gets properties of the tokenizer for the given
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>.
+ :
+ : @param $lang The language of the tokenizer to get the properties of.
+ : @return said properties.
+ : @error err:FTST0009 if <code>$lang</code> is not supported.
+ : tokenization specifically.
+ :)
+declare function ft:tokenizer-properties( $lang as xs:language )
+ as element(ft-schema:tokenizer-properties) external;
+
+(:~
+ : Gets properties of the tokenizer for the
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language";>language</a>
+ : returned by <code>ft:current-lang()</code>.
+ :
+ : @return said properties.
+ : @error err:FTST0009 if <code>ft:current-lang()</code> is not supported.
+ :)
+declare function ft:tokenizer-properties()
+ as element(ft-schema:tokenizer-properties) external;
+
+(:===========================================================================:)
+
+(: vim:set et sw=2 ts=2: :)
=== added file 'modules/com/zorba-xquery/www/modules/full-text.xsd'
--- modules/com/zorba-xquery/www/modules/full-text.xsd 1970-01-01 00:00:00 +0000
+++ modules/com/zorba-xquery/www/modules/full-text.xsd 2012-05-16 22:51:21 +0000
@@ -0,0 +1,134 @@
+<?xml version="1.0"?>
+<!--
+ ! Copyright 2006-2011 The FLWOR Foundation.
+ !
+ ! Licensed under the Apache License, Version 2.0 (the "License");
+ ! you may not use this file except in compliance with the License.
+ ! You may obtain a copy of the License at
+ !
+ ! http://www.apache.org/licenses/LICENSE-2.0
+ !
+ ! Unless required by applicable law or agreed to in writing, software
+ ! distributed under the License is distributed on an "AS IS" BASIS,
+ ! WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ! See the License for the specific language governing permissions and
+ ! limitations under the License.
+-->
+
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema";
+ targetNamespace="http://www.zorba-xquery.com/modules/full-text";
+ xmlns="http://www.zorba-xquery.com/modules/full-text";
+ elementFormDefault="qualified"
+ attributeFormDefault="unqualified">
+
+ <!--======================================================================-->
+
+ <xs:element name="compare-options">
+ <xs:complexType>
+ <xs:attributeGroup ref="compare-attributes"/>
+ </xs:complexType>
+ </xs:element>
+
+ <xs:attributeGroup name="compare-attributes">
+ <xs:attribute name="case" type="sensitivity" default="insensitive"/>
+ <xs:attribute name="diacritics" type="sensitivity" default="insensitive"/>
+ <xs:attribute name="stem" type="yes-no-both" default="no"/>
+ </xs:attributeGroup>
+
+ <xs:simpleType name="sensitivity">
+ <xs:restriction base="xs:string">
+ <xs:enumeration value="insensitive"/>
+ <xs:enumeration value="sensitive"/>
+ <xs:enumeration value="both"/>
+ </xs:restriction>
+ </xs:simpleType>
+
+ <xs:simpleType name="yes-no-both">
+ <xs:restriction base="xs:string">
+ <xs:enumeration value="yes"/>
+ <xs:enumeration value="no"/>
+ <xs:enumeration value="both"/>
+ </xs:restriction>
+ </xs:simpleType>
+
+ <xs:complexType name="boolean-value">
+ <xs:attribute name="value" type="xs:boolean" use="required"/>
+ </xs:complexType>
+
+ <!--======================================================================-->
+
+ <xs:element name="token">
+ <xs:complexType>
+
+ <!-- The language of the token. -->
+ <xs:attribute name="lang" type="xs:language"/>
+
+ <!-- The sentence number. -->
+ <xs:attribute name="sentence" type="xs:nonNegativeInteger" use="required"/>
+
+ <!-- The paragraph number. -->
+ <xs:attribute name="paragraph" type="xs:nonNegativeInteger" use="required"/>
+
+ <!-- The token string value. -->
+ <xs:attribute name="value" type="xs:string" use="required"/>
+
+ <!--
+ ! A reference to the originating node. If the token occurred within an
+ ! element, the reference refers to the text node. If the token occurred
+ ! within an attribute, the reference refers to the attribute node.
+ -->
+ <xs:attribute name="node-ref" type="xs:anyURI"/>
+
+ </xs:complexType>
+ </xs:element>
+
+ <!--======================================================================-->
+
+ <xs:element name="tokenizer-properties">
+ <xs:complexType>
+ <xs:all>
+
+ <!--
+ ! If true, XML comments separate tokens. (No example can be provided
+ ! here because it is illegal to nest an XML comment inside an XML
+ ! comment.)
+ -->
+ <xs:element name="comments-separate-tokens" type="boolean-value"/>
+
+ <!--
+ ! If true, XML elements separate tokens. For example,
+ ! <b>B</b>old would be 2 tokens instead of 1.
+ -->
+ <xs:element name="elements-separate-tokens" type="boolean-value"/>
+
+ <!--
+ ! If true, XML processing instructions separate tokens. For example,
+ ! net<?PI pi?>work would be 2 tokens instead of 1.
+ -->
+ <xs:element name="processing-instructions-separate-tokens" type="boolean-value"/>
+
+ <!--
+ ! The list of languages that the tokenizer can tokenize.
+ -->
+ <xs:element name="supported-languages">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="lang" type="xs:language" maxOccurs="unbounded"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+
+ </xs:all>
+
+ <!--
+ ! The tokenizer's identifying URI.
+ -->
+ <xs:attribute name="uri" type="xs:anyURI"/>
+
+ </xs:complexType>
+ </xs:element>
+
+ <!--======================================================================-->
+
+</xs:schema>
+<!-- vim:set et sw=2 ts=2: -->
=== modified file 'modules/com/zorba-xquery/www/modules/http-client.xq.src/http_request_handler.cpp'
--- modules/com/zorba-xquery/www/modules/http-client.xq.src/http_request_handler.cpp 2012-04-24 12:39:38 +0000
+++ modules/com/zorba-xquery/www/modules/http-client.xq.src/http_request_handler.cpp 2012-05-16 22:51:21 +0000
@@ -39,7 +39,6 @@
theSerStream(NULL),
thePost(NULL),
theLast(NULL),
- theLastSerializerOptions(NULL),
theIsHeadRequest(false)
{
theHeaderLists.push_back(NULL);
@@ -260,6 +259,7 @@
void HttpRequestHandler::cleanUpBody()
{
delete theSerStream;
+ theSerStream = 0;
theLastBodyHadContent = false;
}
=== modified file 'modules/com/zorba-xquery/www/modules/pregenerated/errors.xq'
--- modules/com/zorba-xquery/www/modules/pregenerated/errors.xq 2012-04-24 12:39:38 +0000
+++ modules/com/zorba-xquery/www/modules/pregenerated/errors.xq 2012-05-16 22:51:21 +0000
@@ -188,6 +188,7 @@
(:~
:
+ : The thesaurus data file's endianness does not match that of the CPU.
:
:)
declare variable $zerr:ZXQP8402 as xs:QName := fn:QName($zerr:NS, "zerr:ZXQP8402");
=== modified file 'modules/com/zorba-xquery/www/modules/project_xqdoc.xq'
--- modules/com/zorba-xquery/www/modules/project_xqdoc.xq 2012-04-24 22:08:01 +0000
+++ modules/com/zorba-xquery/www/modules/project_xqdoc.xq 2012-05-16 22:51:21 +0000
@@ -59,7 +59,7 @@
: @return empty sequence.
:)
declare %an:sequential function pxqdoc:delete-XML-dir(
- $xqdocPath as xs:string)
+ $xqdocPath as xs:string) as empty-sequence()
{
variable $xqdocXMLPath := fn:concat( $xqdocPath,
file:directory-separator(),
@@ -92,7 +92,7 @@
:)
declare %an:sequential function pxqdoc:generate-xqdoc-XML(
$zorbaManifestPath as xs:string,
- $xqdocPath as xs:string)
+ $xqdocPath as xs:string) as empty-sequence()
{
(: Note: only the modules that are configured in the Zorba version you are using will be build :)
variable $xqdocXMLPath := concat($xqdocPath, file:directory-separator(), "xml");
=== modified file 'modules/com/zorba-xquery/www/modules/xqdoc2xhtml/index.xq'
--- modules/com/zorba-xquery/www/modules/xqdoc2xhtml/index.xq 2012-04-24 12:39:38 +0000
+++ modules/com/zorba-xquery/www/modules/xqdoc2xhtml/index.xq 2012-05-16 22:51:21 +0000
@@ -268,7 +268,7 @@
:)
declare %an:sequential function xqdoc2html:copy-xhtml-requisites(
$xhtmlRequisitesPath as xs:string,
- $xqdocBuildPath as xs:string)
+ $xqdocBuildPath as xs:string) as empty-sequence()
{
let $xhtmlPath := fn:concat($xqdocBuildPath, file:directory-separator(), "xhtml"),
$xmlPath := fn:concat($xqdocBuildPath, file:directory-separator(), "xml"),
@@ -370,7 +370,7 @@
$xqdocBuildPath as xs:string,
$indexHtmlPath as xs:string,
$zorbaVersion as xs:string,
- $xhtmlRequisitesPath as xs:string)
+ $xhtmlRequisitesPath as xs:string) as empty-sequence()
{
(: fill out $xqdoc2html:ZorbaManifest :)
xqdoc2html:collectZorbaManifestEntries($zorbaManifestPath, $xqdocBuildPath);
@@ -839,9 +839,7 @@
if(fn:matches($specLine, "Args:")) then
let $arg_split := fn:substring-after($specLine, "-x")
return
- if(fn:string-length($arg_split) eq 0) then
- fn:error($err:UE008, fn:concat("Unknown Args: in spec file for example <", $exampleSource,"> .
- Add the example input and expected output by hand in the example, in a commentary that should also include the word 'output'."))
+ if(fn:string-length($arg_split) eq 0) then string-join($specLines, " ")
else
let $var_value := fn:tokenize($arg_split, "=")
let $var_name := fn:normalize-space(fn:replace($var_value[1], ":$", ""))
=== modified file 'modules/org/expath/ns/file.xq'
--- modules/org/expath/ns/file.xq 2012-04-24 12:39:38 +0000
+++ modules/org/expath/ns/file.xq 2012-05-16 22:51:21 +0000
@@ -619,7 +619,7 @@
:)
declare function file:glob-to-regex(
$pattern as xs:string
-) {
+) as xs:string {
let $pattern := fn:replace($pattern, '(\.|\[|\]|\\|/|\||\-|\^|\$|\?|\*|\+|\{|\}|\(|\))','\\$1')
let $pattern := fn:replace($pattern, '\\\?', '.')
let $pattern := fn:replace($pattern, '\\\*', '.*')
=== modified file 'modules/w3c/xpath_functions.xq'
--- modules/w3c/xpath_functions.xq 2012-04-24 12:39:38 +0000
+++ modules/w3c/xpath_functions.xq 2012-05-16 22:51:21 +0000
@@ -522,6 +522,12 @@
declare function fn:parse-xml( $arg as xs:string?, $baseURI as xs:string) as document-node(element(*, xs:untyped)) external;
(:~
+ : @see for semantics please check <a href="http://www.w3.org/TR/xpath-functions-30/#func-parse-xml-fragment";>fn:parse-xml-fragment</a>
+ :)
+declare function fn:parse-xml-fragment( $arg as xs:string?) as document-node(element(*, xs:untyped)) external;
+
+
+(:~
: @see for semantics please check <a href="http://www.w3.org/TR/xpath-functions-30/#func-prefix-from-QName";>fn:prefix-from-QName</a>
:)
declare function fn:prefix-from-QName($arg as xs:QName?) as xs:NCName? external;
@@ -864,6 +870,11 @@
(:~
: @see for semantics please check <a href="http://www.w3.org/TR/xpath-functions-30/#func-has-children";>fn:has-children</a>
:)
+declare function fn:has-children() as xs:boolean external;
+
+(:~
+ : @see for semantics please check <a href="http://www.w3.org/TR/xpath-functions-30/#func-has-children";>fn:has-children</a>
+ :)
declare function fn:has-children($node as node()?) as xs:boolean external;
(:~
@@ -923,12 +934,17 @@
(:~
: @see for semantics please check <a href="http://www.w3.org/TR/xpath-functions-30/#func-namespace-uri";>fn:namespace-uri</a>
- :)
+ :)
declare function fn:namespace-uri($arg as node()?) as xs:anyURI external;
(:~
: @see for semantics please check <a href="http://www.w3.org/TR/xpath-functions-30/#func-nilled";>fn:nilled</a>
:)
+declare function fn:nilled() as xs:boolean external;
+
+(:~
+ : @see for semantics please check <a href="http://www.w3.org/TR/xpath-functions-30/#func-nilled";>fn:nilled</a>
+ :)
declare function fn:nilled($arg as node()?) as xs:boolean? external;
(:~
=== modified file 'scripts/zt-wn-get'
--- scripts/zt-wn-get 2012-04-24 12:39:38 +0000
+++ scripts/zt-wn-get 2012-05-16 22:51:21 +0000
@@ -22,7 +22,7 @@
echo 'Arguments: [--workdir <workdir>] [--builddir <builddir>]'
echo ' [--thesaurusurl <thesaurusurl>]'
echo ' <zorba_repository>'
- echo '<zorba_repository> is the top-level SVN working copy.'
+ echo '<zorba_repository> is the top-level BZR working copy.'
echo '<workdir> is a temp directory to download and unzip XQTS (default: /tmp).'
echo '<builddir> is the directory Zorba has been built in'
echo ' (default: <zorba_repository>/build)'
@@ -71,8 +71,8 @@
echo Build dir is at $BUILD
# Compile thesaurus to binary format
-mkdir -p $BUILD/test/rbkt/thesauri
-THESAURUS_DEST="$BUILD/test/rbkt/thesauri/wordnet-en.zth"
+mkdir -p $BUILD/LIB_PATH/edu/princeton/wordnet
+THESAURUS_DEST="$BUILD/LIB_PATH/edu/princeton/wordnet/wordnet-en.zth"
echo "Compiling thesaurus to $THESAURUS_DEST..."
untar_dir=`mktemp -d "$WORK/thesaurus.XXXXXX"`
cd "$untar_dir"
@@ -83,5 +83,5 @@
rm -rf "$untar_dir"
echo Done.
-echo Now re-run CMake in the build directory to pick up thesaurus tests.
+
# vim:set et sw=2 ts=2:
=== modified file 'src/api/CMakeLists.txt'
--- src/api/CMakeLists.txt 2012-04-24 12:39:38 +0000
+++ src/api/CMakeLists.txt 2012-05-16 22:51:21 +0000
@@ -62,8 +62,9 @@
IF (NOT ZORBA_NO_FULL_TEXT)
LIST(APPEND API_SRCS
stemmer.cpp
- stemmer_wrapper.cpp
- thesaurus.cpp)
+ stemmer_wrappers.cpp
+ thesaurus.cpp
+ thesaurus_wrappers.cpp)
ENDIF (NOT ZORBA_NO_FULL_TEXT)
ADD_SRC_SUBFOLDER(API_SRCS serialization API_SERIALIZATION_SRCS)
=== modified file 'src/api/item_sequence_chainer.h'
--- src/api/item_sequence_chainer.h 2012-04-24 12:39:38 +0000
+++ src/api/item_sequence_chainer.h 2012-05-16 22:51:21 +0000
@@ -17,13 +17,15 @@
#ifndef ZORBA_ITERATOR_ITEM_SEQUENCE_CHAINER_H
#define ZORBA_ITERATOR_ITEM_SEQUENCE_CHAINER_H
+#include <vector>
+#include <set>
+#include <assert.h>
+
#include <zorba/config.h>
#include <zorba/item_sequence.h>
#include <zorba/iterator.h>
#include <zorba/item.h>
#include <zorba/zorba_string.h>
-#include <vector>
-#include <set>
namespace zorba {
@@ -85,7 +87,19 @@
struct Comparator {
bool operator()(const Item& i1, const Item& i2) const
{
- return i1.getStringValue().compare(i2.getStringValue()) < 0;
+ assert(i1.getType().getLocalName() == "QName");
+ assert(i2.getType().getLocalName() == "QName");
+ int lNamespaceComparison = i1.getNamespace().compare(i2.getNamespace());
+ if (lNamespaceComparison < 0)
+ {
+ return true;
+ }
+ if (lNamespaceComparison > 0)
+ {
+ return false;
+ }
+ // Namespaces are equal, comparing local names.
+ return (i1.getLocalName().compare(i2.getLocalName()) < 0);
}
};
=== modified file 'src/api/serialization/serializer.cpp'
--- src/api/serialization/serializer.cpp 2012-04-24 12:39:38 +0000
+++ src/api/serialization/serializer.cpp 2012-05-16 22:51:21 +0000
@@ -100,7 +100,6 @@
}
}
-
////////////////////////////////////////////////////////////////////////////////
// //
// Default emitter //
@@ -194,7 +193,7 @@
unicode::code_point cp = utf8::next_char(temp);
// raise an error iff (1) the serialization format is XML 1.0 and (2) the given character is an invalid XML 1.0 character
- if (ser && ser->method == PARAMETER_VALUE_XML && ser->version == "1.0" && !xml::is_valid(cp))
+ if (ser && ser->method == PARAMETER_VALUE_XML && ser->version == PARAMETER_VALUE_VERSION_1_0 && !xml::is_valid(cp))
throw XQUERY_EXCEPTION( err::FOCH0001, ERROR_PARAMS( cp ) );
if (cp >= 0x10000 && cp <= 0x10FFFF)
@@ -219,7 +218,7 @@
// raise an error iff (1) the serialization format is XML 1.0 and (2) the given character is an invalid XML 1.0 character
if (ser && ser->method == PARAMETER_VALUE_XML &&
- ser->version == "1.0" && !xml::is_valid(static_cast<unsigned>(*chars)))
+ ser->version == PARAMETER_VALUE_VERSION_1_0 && !xml::is_valid(static_cast<unsigned>(*chars)))
throw XQUERY_EXCEPTION(
err::XQST0090,
ERROR_PARAMS( static_cast<unsigned>( *chars ), xml::v1_0 )
@@ -431,7 +430,7 @@
}
else if (item->getNodeKind() == store::StoreConsts::attributeNode)
{
- throw XQUERY_EXCEPTION(err::SENR0001,
+ throw XQUERY_EXCEPTION(err::SENR0001,
ERROR_PARAMS(item->getStringValue(), ZED(AttributeNode)));
}
else
@@ -858,7 +857,7 @@
emitter::emit_declaration();
if (ser->omit_xml_declaration == PARAMETER_VALUE_NO) {
- tr << "<?xml version=\"" << ser->version;
+ tr << "<?xml version=\"" << ser->version_string;
switch (ser->encoding) {
case PARAMETER_VALUE_UTF_8:
case PARAMETER_VALUE_UTF_16:
@@ -1218,7 +1217,7 @@
// an element written as <br/> or <br></br> in an XSLT stylesheet MUST
// be output as <br>.
if (is_html_empty_content_model_element(item) &&
- ztd::equals(ser->version, "4.0", 3))
+ ser->version == PARAMETER_VALUE_VERSION_4_0)
tr << ">";
else
tr << "/>";
@@ -2004,7 +2003,8 @@
undeclare_prefixes = PARAMETER_VALUE_NO;
- version = "1.0";
+ version = PARAMETER_VALUE_VERSION_1_0;
+ version_string = "1.0";
version_has_default_value = true;
indent = PARAMETER_VALUE_NO;
@@ -2119,8 +2119,18 @@
}
else if (!strcmp(aName, "version"))
{
- version = aValue;
+ version_string = aValue;
version_has_default_value = false;
+ if (version_string == "1.0")
+ version = PARAMETER_VALUE_VERSION_1_0;
+ else if (version_string == "1.1")
+ version = PARAMETER_VALUE_VERSION_1_1;
+ else if (version_string == "4.0")
+ version = PARAMETER_VALUE_VERSION_4_0;
+ else if (version_string == "4.01")
+ version = PARAMETER_VALUE_VERSION_4_01;
+ else
+ version = PARAMETER_VALUE_VERSION_OTHER;
}
else if (!strcmp(aName, "doctype-system"))
{
@@ -2156,51 +2166,52 @@
void
serializer::validate_parameters(void)
{
- if (method == PARAMETER_VALUE_XML || method == PARAMETER_VALUE_XHTML)
+ if (method == PARAMETER_VALUE_XML || method == PARAMETER_VALUE_XHTML)
{
// XML-only validation
- if (method == PARAMETER_VALUE_XML)
+ if (method == PARAMETER_VALUE_XML)
{
- if (version != "1.0" && version != "1.1")
+ if (version != PARAMETER_VALUE_VERSION_1_0 && version != PARAMETER_VALUE_VERSION_1_1)
throw XQUERY_EXCEPTION(
err::SESU0013, ERROR_PARAMS( version, "XML", "\"1.0\", \"1.1\"" )
);
}
// XHTML-only validation
- if (method == PARAMETER_VALUE_XHTML)
+ if (method == PARAMETER_VALUE_XHTML)
{
}
// XML and XHTML validation
- if (omit_xml_declaration == PARAMETER_VALUE_YES)
+ if (omit_xml_declaration == PARAMETER_VALUE_YES)
{
if (standalone != PARAMETER_VALUE_OMIT)
throw XQUERY_EXCEPTION(
err::SEPM0009, ERROR_PARAMS( ZED( SEPM0009_NotOmit ) )
);
- if (version != "1.0" && !doctype_system.empty())
+ if (version != PARAMETER_VALUE_VERSION_1_0 && !doctype_system.empty())
throw XQUERY_EXCEPTION(
err::SEPM0009, ERROR_PARAMS( ZED( SEPM0009_Not10 ) )
);
}
- if (undeclare_prefixes == PARAMETER_VALUE_YES && version == "1.0")
+ if (undeclare_prefixes == PARAMETER_VALUE_YES && version == PARAMETER_VALUE_VERSION_1_0)
throw XQUERY_EXCEPTION( err::SEPM0010 );
}
- if (method == PARAMETER_VALUE_HTML)
+ if (method == PARAMETER_VALUE_HTML)
{
// Default value for "version" when method is HTML is "4.0"
- if (version_has_default_value)
+ if (version_has_default_value)
{
- version = "4.0";
+ version = PARAMETER_VALUE_VERSION_4_0;
+ version_string = "4.0";
}
- else if (!(ztd::equals(version, "4.0", 3) || ztd::equals(version, "4.01", 4)))
+ else if (version != PARAMETER_VALUE_VERSION_4_0 && version != PARAMETER_VALUE_VERSION_4_01)
{
throw XQUERY_EXCEPTION(
- err::SESU0013, ERROR_PARAMS( version, "HTML", "\"4.0\", \"4.01\"" )
+ err::SESU0013, ERROR_PARAMS( version_string, "HTML", "\"4.0\", \"4.01\"" )
);
}
}
@@ -2278,13 +2289,13 @@
validate_parameters();
- if (!setup(aOStream))
+ if (!setup(aOStream))
{
return;
}
// in case we use SAX event notifications
- if (aHandler)
+ if (aHandler)
{
// only allow XML-based methods for SAX notifications
if (method != PARAMETER_VALUE_XML &&
@@ -2303,10 +2314,10 @@
store::Item_t lItem;
//+ aObject->open();
- while (aObject->next(lItem))
+ while (aObject->next(lItem))
{
// PUL's cannot be serialized
- if (lItem->isPul())
+ if (lItem->isPul())
{
throw ZORBA_EXCEPTION(zerr::ZAPI0007_CANNOT_SERIALIZE_PUL);
}
@@ -2329,7 +2340,7 @@
{
validate_parameters();
- if (!setup(stream))
+ if (!setup(stream))
{
return;
}
@@ -2338,27 +2349,27 @@
store::Item_t lItem;
//object->open();
- while (object->next(lItem))
+ while (object->next(lItem))
{
Zorba_SerializerOptions_t* lSerParams = aHandler(aHandlerData);
- if (lSerParams)
+ if (lSerParams)
{
SerializerImpl::setSerializationParameters(*this, *lSerParams);
- if (!setup(stream))
+ if (!setup(stream))
{
return;
}
}
// PUL's cannot be serialized
- if (lItem->isPul())
+ if (lItem->isPul())
{
throw ZORBA_EXCEPTION(zerr::ZAPI0007_CANNOT_SERIALIZE_PUL);
}
e->emit_item(&*lItem);
}
-
+
//object->close();
e->emit_declaration_end();
}
=== modified file 'src/api/serialization/serializer.h'
--- src/api/serialization/serializer.h 2012-04-24 12:39:38 +0000
+++ src/api/serialization/serializer.h 2012-05-16 22:51:21 +0000
@@ -71,7 +71,15 @@
PARAMETER_VALUE_BINARY,
PARAMETER_VALUE_UTF_8,
- PARAMETER_VALUE_UTF_16
+ PARAMETER_VALUE_UTF_16,
+
+ // Values for the XML/HTML version
+ PARAMETER_VALUE_VERSION_1_0, // used for XML 1.0
+ PARAMETER_VALUE_VERSION_1_1, // used for XML 1.1
+ PARAMETER_VALUE_VERSION_4_0, // used for HTML 4.0
+ PARAMETER_VALUE_VERSION_4_01, // used for HTML 4.01
+ PARAMETER_VALUE_VERSION_OTHER // given by the version string
+
} PARAMETER_VALUE_TYPE;
protected:
@@ -98,7 +106,8 @@
short int standalone; // implemented
short int undeclare_prefixes; // "yes" or "no", implemented
void* use_character_maps; // TODO: list of pairs
- zstring version; // "1.0"
+ short int version; // "1.0"
+ zstring version_string; // this the version as a string
short int indent; // "yes" or "no", implemented
bool version_has_default_value; // Used during validation to set version to
// "4.0" when output method is "html"
@@ -251,7 +260,7 @@
/**
* Serializes the given string, performing character expansion
- * if necessary.
+ * if necessary.
*
* @return returns the number of bytes that have not been written. This
* info is used only for streamable items expansion.
=== modified file 'src/api/staticcontextimpl.cpp'
--- src/api/staticcontextimpl.cpp 2012-04-24 12:39:38 +0000
+++ src/api/staticcontextimpl.cpp 2012-05-16 22:51:21 +0000
@@ -42,8 +42,8 @@
#include "context/static_context.h"
#include "context/static_context_consts.h"
#ifndef ZORBA_NO_FULL_TEXT
-#include "context/stemmer_wrappers.h"
-#include "context/thesaurus_wrappers.h"
+#include "stemmer_wrappers.h"
+#include "thesaurus_wrappers.h"
#endif /* ZORBA_NO_FULL_TEXT */
#include "uri_resolver_wrappers.h"
@@ -65,7 +65,6 @@
namespace zorba {
-
/*******************************************************************************
Create a StaticContextImpl obj as well as an internal static_context obj S.
S is created as a child of the zorba root sctx. This constructor is used
@@ -74,6 +73,7 @@
********************************************************************************/
StaticContextImpl::StaticContextImpl(DiagnosticHandler* aDiagnosticHandler)
:
+ theMaxVarId(2),
theDiagnosticHandler(aDiagnosticHandler),
theUserDiagnosticHandler(true),
theCollectionMgr(0)
@@ -98,6 +98,7 @@
DiagnosticHandler* aDiagnosticHandler)
:
theCtx(aCtx),
+ theMaxVarId(2),
theDiagnosticHandler(aDiagnosticHandler),
theUserDiagnosticHandler(true),
theCollectionMgr(0)
@@ -119,6 +120,7 @@
StaticContextImpl::StaticContextImpl(const StaticContextImpl& aStaticContext)
:
StaticContext(),
+ theMaxVarId(2),
theDiagnosticHandler(aStaticContext.theDiagnosticHandler),
theUserDiagnosticHandler(aStaticContext.theUserDiagnosticHandler),
theCollectionMgr(0)
@@ -1024,9 +1026,11 @@
theSctxMap = impl.theCompilerCB->theSctxMap;
}
+
static void
-toInternalPath(const std::vector<String>& aPublicStrings,
- std::vector<zstring>& aInternalStrings)
+toInternalPath(
+ const std::vector<String>& aPublicStrings,
+ std::vector<zstring>& aInternalStrings)
{
for (std::vector<String>::const_iterator lIter = aPublicStrings.begin();
lIter != aPublicStrings.end(); ++lIter)
@@ -1043,9 +1047,11 @@
}
}
+
static void
-toPublicPath(const std::vector<zstring>& aInternalStrings,
- std::vector<String>& aPublicStrings)
+toPublicPath(
+ const std::vector<zstring>& aInternalStrings,
+ std::vector<String>& aPublicStrings)
{
for (std::vector<zstring>::const_iterator lIter = aInternalStrings.begin();
lIter != aInternalStrings.end(); ++lIter)
@@ -1054,6 +1060,7 @@
}
}
+
void
StaticContextImpl::setURIPath(const std::vector<String> &aURIPath)
{
=== modified file 'src/api/staticcontextimpl.h'
--- src/api/staticcontextimpl.h 2012-04-24 12:39:38 +0000
+++ src/api/staticcontextimpl.h 2012-05-16 22:51:21 +0000
@@ -33,16 +33,29 @@
/*******************************************************************************
- theCtx : rchandle to the internal static_context obj that is
- wrapped by "this".
-
- theSctxMap :
-
- theDiagnosticHandler : Pointer to an error handle to handle any errors raised
- by the methods of this class.
- theUserDiagnosticHandler : If true, theDiagnosticHandler is owned by "this", and it
- should be deleted when "this" is destroyed. Otherwise,
- theDiagnosticHandler is not owned by "this".
+ theCtx :
+ --------
+ rchandle to the internal static_context obj that is wrapped by "this".
+
+ theSctxMap :
+ ------------
+
+ theMaxVarId :
+ -------------
+ If loadProlog() is called on "this", the compiler will store in theMaxVarId
+ the max variable id generated by the compilation of the given prolog. This
+ is needed by the "outer" query, where the prolog will be loaded to.
+
+ theDiagnosticHandler :
+ ----------------------
+ Pointer to an error handle to handle any errors raised by the methods of
+ this class.
+
+ theUserDiagnosticHandler :
+ --------------------------
+ If true, theDiagnosticHandler is owned by "this", and it should be deleted
+ when "this" is destroyed. Otherwise, theDiagnosticHandler is not owned by
+ "this".
********************************************************************************/
class StaticContextImpl : public StaticContext
{
@@ -54,6 +67,8 @@
std::map<int, static_context_t> theSctxMap;
+ ulong theMaxVarId;
+
DiagnosticHandler * theDiagnosticHandler;
bool theUserDiagnosticHandler;
@@ -70,25 +85,27 @@
virtual ~StaticContextImpl();
- void loadProlog(const String&, const Zorba_CompilerHints_t &hints);
+ void loadProlog(const String&, const Zorba_CompilerHints_t& hints);
+
+ ulong getMaxVarId() const { return theMaxVarId; }
+
+ void setMaxVarId(ulong v) { if (v > theMaxVarId) theMaxVarId = v; }
StaticContext_t createChildContext() const;
- bool addNamespace( const String& prefix, const String& URI );
-
- String getNamespaceURIByPrefix( const String& prefix ) const;
-
- void
- getNamespaceBindings( NsBindings& aBindings ) const;
-
- bool setDefaultElementAndTypeNamespace( const String& URI );
-
- String getDefaultElementAndTypeNamespace( ) const;
-
- bool setDefaultFunctionNamespace( const String& URI );
-
- virtual String
- getDefaultFunctionNamespace( ) const;
+ bool addNamespace(const String& prefix, const String& URI);
+
+ String getNamespaceURIByPrefix(const String& prefix) const;
+
+ void getNamespaceBindings(NsBindings& aBindings) const;
+
+ bool setDefaultElementAndTypeNamespace(const String& URI);
+
+ String getDefaultElementAndTypeNamespace() const;
+
+ bool setDefaultFunctionNamespace(const String& URI);
+
+ virtual String getDefaultFunctionNamespace() const;
virtual void
addCollation( const String& URI );
=== renamed file 'src/api/stemmer_wrapper.cpp' => 'src/api/stemmer_wrappers.cpp'
--- src/api/stemmer_wrapper.cpp 2012-04-24 12:39:38 +0000
+++ src/api/stemmer_wrappers.cpp 2012-05-16 22:51:21 +0000
@@ -23,7 +23,7 @@
#include "diagnostics/assert.h"
#include "util/cxx_util.h"
-#include "stemmer_wrapper.h"
+#include "stemmer_wrappers.h"
using namespace zorba::locale;
@@ -32,8 +32,8 @@
///////////////////////////////////////////////////////////////////////////////
-StemmerWrapper::StemmerWrapper( zorba::Stemmer::ptr p ) :
- api_stemmer_( std::move( p ) )
+StemmerWrapper::StemmerWrapper( zorba::Stemmer::ptr api_stemmer ) :
+ api_stemmer_( std::move( api_stemmer ) )
{
ZORBA_ASSERT( api_stemmer_.get() );
}
@@ -42,6 +42,12 @@
api_stemmer_.release()->destroy();
}
+void StemmerWrapper::properties( Properties *props ) const {
+ zorba::Stemmer::Properties api_props;
+ api_stemmer_->properties( &api_props );
+ props->uri = api_props.uri;
+}
+
void StemmerWrapper::stem( zstring const &word, iso639_1::type lang,
zstring *result ) const {
String const api_word( Unmarshaller::newString( word ) );
@@ -52,19 +58,22 @@
///////////////////////////////////////////////////////////////////////////////
StemmerProviderWrapper::
-StemmerProviderWrapper( zorba::StemmerProvider const *p ) :
- api_stemmer_provider_( p )
+StemmerProviderWrapper( zorba::StemmerProvider const *api_stemmer_provider ) :
+ api_stemmer_provider_( api_stemmer_provider )
{
ZORBA_ASSERT( api_stemmer_provider_ );
}
-Stemmer::ptr
-StemmerProviderWrapper::get_stemmer( iso639_1::type lang ) const {
- zorba::Stemmer::ptr p( api_stemmer_provider_->getStemmer( lang ) );
- Stemmer::ptr result;
- if ( p.get() )
- result.reset( new StemmerWrapper( std::move( p ) ) );
- return std::move( result );
+bool StemmerProviderWrapper::getStemmer( iso639_1::type lang,
+ Stemmer::ptr *result ) const {
+ zorba::Stemmer::ptr api_ptr;
+ zorba::Stemmer::ptr *const api_ptr_ptr = result ? &api_ptr : nullptr;
+ if ( api_stemmer_provider_->getStemmer( lang, api_ptr_ptr ) ) {
+ if ( result )
+ result->reset( new StemmerWrapper( std::move( api_ptr ) ) );
+ return true;
+ }
+ return false;
}
///////////////////////////////////////////////////////////////////////////////
=== renamed file 'src/api/stemmer_wrapper.h' => 'src/api/stemmer_wrappers.h'
--- src/api/stemmer_wrapper.h 2012-04-24 12:39:38 +0000
+++ src/api/stemmer_wrappers.h 2012-05-16 22:51:21 +0000
@@ -35,6 +35,7 @@
// inherited
void destroy() const;
+ void properties( Properties* ) const;
void stem( zstring const &word, locale::iso639_1::type lang,
zstring *result ) const;
private:
@@ -50,7 +51,7 @@
}
// inherited
- Stemmer::ptr get_stemmer( locale::iso639_1::type lang ) const;
+ bool getStemmer( locale::iso639_1::type, Stemmer::ptr* = 0 ) const;
private:
zorba::StemmerProvider const *const api_stemmer_provider_;
};
=== modified file 'src/api/thesaurus.cpp'
--- src/api/thesaurus.cpp 2012-04-24 12:39:38 +0000
+++ src/api/thesaurus.cpp 2012-05-16 22:51:21 +0000
@@ -25,9 +25,11 @@
// out-of-line since it's virtual
}
-//Thesaurus::iterator::~iterator() {
-// // out-of-line since it's virtual
-//}
+#if 0
+Thesaurus::iterator::~iterator() {
+ // out-of-line since it's virtual
+}
+#endif
///////////////////////////////////////////////////////////////////////////////
=== renamed file 'src/context/thesaurus_wrappers.cpp' => 'src/api/thesaurus_wrappers.cpp'
--- src/context/thesaurus_wrappers.cpp 2012-04-24 12:39:38 +0000
+++ src/api/thesaurus_wrappers.cpp 2012-05-16 22:51:21 +0000
@@ -87,6 +87,27 @@
///////////////////////////////////////////////////////////////////////////////
+ThesaurusProviderWrapper::
+ThesaurusProviderWrapper( zorba::ThesaurusProvider const *p ) :
+ api_thesaurus_provider_( p )
+{
+ ZORBA_ASSERT( api_thesaurus_provider_ );
+}
+
+bool ThesaurusProviderWrapper::getThesaurus( iso639_1::type lang,
+ Thesaurus::ptr *result ) const {
+ zorba::Thesaurus::ptr api_ptr;
+ zorba::Thesaurus::ptr *const api_ptr_ptr = result ? &api_ptr : nullptr;
+ if ( api_thesaurus_provider_->getThesaurus( lang, api_ptr_ptr ) ) {
+ if ( result )
+ result->reset( new ThesaurusWrapper( std::move( api_ptr ) ) );
+ return true;
+ }
+ return false;
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
} // namespace internal
} // namespace zorba
=== renamed file 'src/context/thesaurus_wrappers.h' => 'src/api/thesaurus_wrappers.h'
--- src/context/thesaurus_wrappers.h 2012-04-24 12:39:38 +0000
+++ src/api/thesaurus_wrappers.h 2012-05-16 22:51:21 +0000
@@ -22,6 +22,7 @@
#ifndef ZORBA_NO_FULL_TEXT
#include <zorba/thesaurus.h>
+
#include "runtime/full_text/thesaurus.h"
namespace zorba {
@@ -54,6 +55,17 @@
zorba::Thesaurus::ptr api_thesaurus_;
};
+class ThesaurusProviderWrapper : public ThesaurusProvider {
+public:
+ ThesaurusProviderWrapper( zorba::ThesaurusProvider const* );
+
+ // inherited
+ bool getThesaurus( locale::iso639_1::type, Thesaurus::ptr* ) const;
+
+private:
+ zorba::ThesaurusProvider::ptr const api_thesaurus_provider_;
+};
+
///////////////////////////////////////////////////////////////////////////////
} // namespace internal
=== modified file 'src/api/uri_resolver_wrappers.cpp'
--- src/api/uri_resolver_wrappers.cpp 2012-04-24 12:39:38 +0000
+++ src/api/uri_resolver_wrappers.cpp 2012-05-16 22:51:21 +0000
@@ -15,24 +15,20 @@
*/
#include "stdafx.h"
+#include <zorba/thesaurus.h>
+
+#include "runtime/full_text/thesaurus.h"
+
+#include "thesaurus_wrappers.h"
+#include "unmarshaller.h"
#include "uri_resolver_wrappers.h"
#include "uriresolverimpl.h"
-#include "unmarshaller.h"
-#include <zorba/thesaurus.h>
-#include <runtime/full_text/thesaurus.h>
-#include <context/thesaurus_wrappers.h>
namespace zorba
{
// "Convenience" class for passing an internal EntityData object to
- // external mappers/resolvers. This can serve as a plain EntityData or
- // a ThesaurusEntityData. However, when there's another EntityData subclass
- // in future, this won't work as EntityData becomes an ambiguous base class...
-#ifndef ZORBA_NO_FULL_TEXT
- class EntityDataWrapper : public ThesaurusEntityData
-#else
+ // external mappers/resolvers.
class EntityDataWrapper : public EntityData
-#endif /* ZORBA_NO_FULL_TEXT */
{
public:
static EntityDataWrapper const* create(internal::EntityData const* aData) {
@@ -45,12 +41,7 @@
return new EntityDataWrapper(EntityData::SCHEMA);
#ifndef ZORBA_NO_FULL_TEXT
case internal::EntityData::THESAURUS:
- {
- EntityDataWrapper* retval = new EntityDataWrapper(EntityData::THESAURUS);
- retval->theThesaurusLang =
- dynamic_cast<const internal::ThesaurusEntityData*>(aData)->getLanguage();
- return retval;
- }
+ return new EntityDataWrapper(EntityData::THESAURUS);
case internal::EntityData::STOP_WORDS:
return new EntityDataWrapper(EntityData::STOP_WORDS);
#endif /* ZORBA_NO_FULL_TEXT */
@@ -67,21 +58,12 @@
return theKind;
}
-#ifndef ZORBA_NO_FULL_TEXT
- virtual zorba::locale::iso639_1::type getLanguage() const {
- return theThesaurusLang;
- }
-#endif /* ZORBA_NO_FULL_TEXT */
-
private:
EntityDataWrapper(EntityData::Kind aKind)
: theKind(aKind)
{}
EntityData::Kind const theKind;
-#ifndef ZORBA_NO_FULL_TEXT
- zorba::locale::iso639_1::type theThesaurusLang;
-#endif /* ZORBA_NO_FULL_TEXT */
};
URIMapperWrapper::URIMapperWrapper(zorba::URIMapper& aUserMapper)
@@ -169,13 +151,13 @@
}
#ifndef ZORBA_NO_FULL_TEXT
else {
- Thesaurus* lUserThesaurus = dynamic_cast<Thesaurus*>(lUserPtr.get());
- if (lUserThesaurus != NULL) {
- // Here we pass memory ownership of the actual Thesaurus to the
- // internal ThesaurusWrapper.
- lRetval = new internal::ThesaurusWrapper
- (Thesaurus::ptr(lUserThesaurus));
- lUserPtr.release();
+ ThesaurusProvider* lUserThesaurusProvider =
+ dynamic_cast<ThesaurusProvider*>(lUserPtr.get());
+ if (lUserThesaurusProvider) {
+ // Here we pass memory ownership of the actual ThesaurusProvider to
+ // the internal ThesaurusWrapper.
+ lRetval = new internal::ThesaurusProviderWrapper
+ (lUserThesaurusProvider);
}
else {
assert(false);
=== modified file 'src/api/xmldatamanagerimpl.cpp'
--- src/api/xmldatamanagerimpl.cpp 2012-04-24 12:39:38 +0000
+++ src/api/xmldatamanagerimpl.cpp 2012-05-16 22:51:21 +0000
@@ -33,21 +33,18 @@
#include "api/staticcontextimpl.h"
#include "api/documentmanagerimpl.h"
#include "api/collectionmanagerimpl.h"
-
+#include "context/static_context.h"
#include "diagnostics/xquery_diagnostics.h"
-
+#include "runtime/util/flowctl_exception.h"
#include "store/api/collection.h"
#include "store/api/item.h"
-#include "system/globalenv.h"
#include "store/api/store.h"
#include "store/api/item_factory.h"
-
-#include "context/static_context.h"
-
-#include "runtime/util/flowctl_exception.h"
+#include "system/globalenv.h"
+#include "zorbamisc/ns_consts.h"
#ifndef ZORBA_NO_FULL_TEXT
-#include "stemmer_wrapper.h"
+#include "stemmer_wrappers.h"
#endif /* ZORBA_NO_FULL_TEXT */
namespace zorba {
@@ -292,7 +289,7 @@
Item empty_item;
Item validated_options;
NsBindings nsPairs;
- Item untyped_type = theFactory->createQName("http://www.w3.org/2001/XMLSchema";, "xs", "untyped");
+ Item untyped_type = theFactory->createQName(XML_SCHEMA_NS, XML_SCHEMA_PREFIX, "untyped");
Item options_node = theFactory->createElementNode(empty_item,
theFactory->createQName(static_context::ZORBA_XML_FN_OPTIONS_NS, "options"),
untyped_type, false, false, nsPairs);
@@ -340,7 +337,7 @@
Item empty_item;
Item validated_options;
NsBindings nsPairs;
- Item untyped_type = theFactory->createQName("http://www.w3.org/2001/XMLSchema";, "xs", "untyped");
+ Item untyped_type = theFactory->createQName(XML_SCHEMA_NS, XML_SCHEMA_PREFIX, "untyped");
Item options_node = theFactory->createElementNode(empty_item,
theFactory->createQName(static_context::ZORBA_XML_FN_OPTIONS_NS, "options"),
untyped_type, false, false, nsPairs);
=== modified file 'src/api/xmldatamanagerimpl.h'
--- src/api/xmldatamanagerimpl.h 2012-04-24 12:39:38 +0000
+++ src/api/xmldatamanagerimpl.h 2012-05-16 22:51:21 +0000
@@ -27,7 +27,7 @@
#include "util/singleton.h"
#ifndef ZORBA_NO_FULL_TEXT
-#include "stemmer_wrapper.h"
+#include "stemmer_wrappers.h"
#endif /* ZORBA_NO_FULL_TEXT */
namespace zorba {
=== modified file 'src/api/xqueryimpl.cpp'
--- src/api/xqueryimpl.cpp 2012-04-24 12:39:38 +0000
+++ src/api/xqueryimpl.cpp 2012-05-16 22:51:21 +0000
@@ -435,7 +435,11 @@
std::istringstream lQueryStream(aQuery.c_str());
- doCompile(lQueryStream, aHints);
+ // 0 is reserved as an invalid var id, and 1 is taken by the context item
+ // in the main module.
+ ulong nextVarId = 2;
+
+ doCompile(lQueryStream, aHints, true, nextVarId);
}
QUERY_CATCH
}
@@ -455,7 +459,11 @@
checkNotClosed();
checkNotCompiled();
- doCompile(aQuery, aHints);
+ // 0 is reserved as an invalid var id, and 1 is taken by the context item
+ // in the main module.
+ ulong nextVarId = 2;
+
+ doCompile(aQuery, aHints, true, nextVarId);
}
QUERY_CATCH
}
@@ -485,9 +493,12 @@
theCompilerCB->theSctxMap =
static_cast<StaticContextImpl*>(aStaticContext.get())->theSctxMap;
+ ulong nextVarId =
+ static_cast<StaticContextImpl*>(aStaticContext.get())->getMaxVarId();
+
std::istringstream lQueryStream(aQuery.c_str());
- doCompile(lQueryStream, aHints);
+ doCompile(lQueryStream, aHints, true, nextVarId);
}
QUERY_CATCH
}
@@ -515,7 +526,10 @@
theCompilerCB->theSctxMap =
static_cast<StaticContextImpl*>(aStaticContext.get())->theSctxMap;
- doCompile(aQuery, aHints);
+ ulong nextVarId =
+ static_cast<StaticContextImpl*>(aStaticContext.get())->getMaxVarId();
+
+ doCompile(aQuery, aHints, true, nextVarId);
}
QUERY_CATCH
}
@@ -527,7 +541,8 @@
void XQueryImpl::doCompile(
std::istream& aQuery,
const Zorba_CompilerHints_t& aHints,
- bool fork_sctx)
+ bool fork_sctx,
+ ulong& nextDynamicVarId)
{
if ( ! theStaticContext )
{
@@ -544,7 +559,9 @@
// otherwise, unless this is a load-prolog query, create a child and we have
// ownership over that one
if (fork_sctx)
+ {
theStaticContext = theStaticContext->create_child_context();
+ }
}
zstring url;
@@ -581,10 +598,6 @@
}
#endif
- // 0 is reserved as an invalid var id, and 1 is taken by the context item
- // in the main module.
- ulong nextDynamicVarId = 2;
-
PlanIter_t planRoot = lCompiler.compile(aQuery, theFileName, nextDynamicVarId);
thePlanProxy = new PlanProxy(planRoot);
@@ -616,7 +629,13 @@
theCompilerCB->setLoadPrologQuery();
- doCompile(lQueryStream, aHints, false);
+ StaticContextImpl* sctx = static_cast<StaticContextImpl*>(aStaticContext.get());
+
+ ulong nextVarId = sctx->getMaxVarId();
+
+ doCompile(lQueryStream, aHints, false, nextVarId);
+
+ sctx->setMaxVarId(nextVarId);
}
QUERY_CATCH
}
=== modified file 'src/api/xqueryimpl.h'
--- src/api/xqueryimpl.h 2012-04-24 12:39:38 +0000
+++ src/api/xqueryimpl.h 2012-05-16 22:51:21 +0000
@@ -356,7 +356,8 @@
void doCompile(
std::istream&,
const Zorba_CompilerHints_t& aHints,
- bool fork_sctx = true);
+ bool fork_sctx,
+ ulong& nextVarId);
PlanWrapper_t generateWrapper();
=== modified file 'src/compiler/codegen/plan_visitor.cpp'
--- src/compiler/codegen/plan_visitor.cpp 2012-04-24 12:39:38 +0000
+++ src/compiler/codegen/plan_visitor.cpp 2012-05-16 22:51:21 +0000
@@ -250,7 +250,7 @@
class plan_ftnode_visitor : public ftnode_visitor
{
public:
- typedef std::list<PlanIter_t> PlanIter_list_t;
+ typedef std::vector<PlanIter_t> PlanIter_list_t;
plan_ftnode_visitor( plan_visitor* v ) : plan_visitor_( v ) { }
=== modified file 'src/compiler/expression/expr_put.cpp'
--- src/compiler/expression/expr_put.cpp 2012-04-24 12:39:38 +0000
+++ src/compiler/expression/expr_put.cpp 2012-05-16 22:51:21 +0000
@@ -41,6 +41,7 @@
#include "compiler/expression/function_item_expr.h"
#include "compiler/parser/parse_constants.h"
+#include "diagnostics/assert.h"
#include "functions/function.h"
#include "functions/udf.h"
=== modified file 'src/compiler/parser/xquery_parser.y'
--- src/compiler/parser/xquery_parser.y 2012-04-24 12:39:38 +0000
+++ src/compiler/parser/xquery_parser.y 2012-05-16 22:51:21 +0000
@@ -2924,12 +2924,42 @@
GroupSpec :
DOLLAR QNAME
{
- $$ = new GroupSpec(LOC(@$), static_cast<QName*>($2), NULL);
+ $$ = new GroupSpec(LOC(@$), static_cast<QName*>($2), NULL, NULL, NULL);
+ }
+ | DOLLAR QNAME GETS ExprSingle
+ {
+ $$ = new GroupSpec(LOC(@$), static_cast<QName*>($2), NULL, $4, NULL);
+ }
+ | DOLLAR QNAME TypeDeclaration GETS ExprSingle
+ {
+ $$ = new GroupSpec(LOC(@$),
+ static_cast<QName*>($2),
+ dynamic_cast<SequenceType*>($3),
+ $5,
+ NULL);
+ }
+ | DOLLAR QNAME TypeDeclaration GETS ExprSingle GroupCollationSpec
+ {
+ $$ = new GroupSpec(LOC(@$),
+ static_cast<QName*>($2),
+ dynamic_cast<SequenceType*>($3),
+ $5,
+ dynamic_cast<GroupCollationSpec*>($6));
+ }
+ | DOLLAR QNAME GETS ExprSingle GroupCollationSpec
+ {
+ $$ = new GroupSpec(LOC(@$),
+ static_cast<QName*>($2),
+ NULL,
+ $4,
+ dynamic_cast<GroupCollationSpec*>($5));
}
| DOLLAR QNAME GroupCollationSpec
{
$$ = new GroupSpec(LOC(@$),
static_cast<QName*>($2),
+ NULL,
+ NULL,
dynamic_cast<GroupCollationSpec*>($3));
}
;
=== modified file 'src/compiler/parsetree/parsenode_print_xqdoc_visitor.cpp'
--- src/compiler/parsetree/parsenode_print_xqdoc_visitor.cpp 2012-04-24 12:39:38 +0000
+++ src/compiler/parsetree/parsenode_print_xqdoc_visitor.cpp 2012-05-16 22:51:21 +0000
@@ -38,6 +38,8 @@
#include "store/api/iterator.h"
#include "system/globalenv.h"
+#include "zorbamisc/ns_consts.h"
+
using namespace std;
@@ -499,10 +501,10 @@
theVersion("1.0"),
theFactory(GENV_ITEMFACTORY)
{
- theNamespaceMap["fn"] = "http://www.w3.org/2005/xpath-functions";;
- theNamespaceMap[""] = "http://www.w3.org/2005/xpath-functions";;
- theNamespaceMap["xs"] = "http://www.w3.org/2001/XMLSchema";;
- theNamespaceMap["local"] = "http://www.w3.org/2005/xquery-local-functions";;
+ theNamespaceMap["fn"] = XQUERY_XPATH_FN_NS;
+ theNamespaceMap[""] = XQUERY_XPATH_FN_NS;
+ theNamespaceMap[XML_SCHEMA_PREFIX] = XML_SCHEMA_NS;
+ theNamespaceMap["local"] = XQUERY_LOCAL_FN_NS;
}
=== modified file 'src/compiler/parsetree/parsenodes.cpp'
--- src/compiler/parsetree/parsenodes.cpp 2012-04-24 12:39:38 +0000
+++ src/compiler/parsetree/parsenodes.cpp 2012-05-16 22:51:21 +0000
@@ -1774,16 +1774,28 @@
vector<rchandle<GroupSpec> >::const_iterator ite = theSpecs.begin();
vector<rchandle<GroupSpec> >::const_iterator end = theSpecs.end();
- for (; ite != end; ++ite)
+ //If no expression is given, then it'll just be the variable.
+ //If this is the case, and the variable is already defined as a
+ //grouping variable, it makes no sense to add it, so this is a
+ //quick optimization.
+ if(spec->get_var_expr() == NULL)
{
- const GroupSpec* currSpec = (*ite).getp();
-
- if (*currSpec->get_var_name() == *spec->get_var_name())
- break;
+ for (; ite != end; ++ite)
+ {
+ const GroupSpec* currSpec = (*ite).getp();
+
+ if (*currSpec->get_var_name() == *spec->get_var_name() &&
+ currSpec->get_var_expr() == NULL)
+ break;
+ }
+
+ if (ite == end)
+ theSpecs.push_back(spec);
}
-
- if (ite == end)
+ else
+ {
theSpecs.push_back(spec);
+ }
}
@@ -1805,10 +1817,14 @@
GroupSpec::GroupSpec(
const QueryLoc& loc_,
rchandle<QName> _var_name_h,
+ rchandle<SequenceType> _var_type_h,
+ rchandle<exprnode> _var_value_h,
rchandle<GroupCollationSpec> _group_coll_spec_h)
:
parsenode(loc_),
var_name_h(_var_name_h),
+ var_type_h(_var_type_h),
+ var_init_expr_h(_var_value_h),
group_coll_spec_h(_group_coll_spec_h)
{
}
@@ -1817,7 +1833,9 @@
void GroupSpec::accept(parsenode_visitor& v) const
{
BEGIN_VISITOR();
+ ACCEPT (var_type_h);
ACCEPT (group_coll_spec_h);
+ ACCEPT (var_init_expr_h);
END_VISITOR();
}
=== modified file 'src/compiler/parsetree/parsenodes.h'
--- src/compiler/parsetree/parsenodes.h 2012-04-24 12:39:38 +0000
+++ src/compiler/parsetree/parsenodes.h 2012-05-16 22:51:21 +0000
@@ -2207,22 +2207,30 @@
/*******************************************************************************
- GroupSpec ::= "$" VarName ("collation" URILiteral)?
+ GroupSpec ::= "$" VarName (TypeDeclaration? ":=" ExprSingle)? ("collation" URILiteral)?
********************************************************************************/
class GroupSpec : public parsenode
{
protected:
rchandle<QName> var_name_h;
rchandle<GroupCollationSpec> group_coll_spec_h;
+ rchandle<exprnode> var_init_expr_h;
+ rchandle<SequenceType> var_type_h;
public:
GroupSpec(
const QueryLoc&,
rchandle<QName>,
+ rchandle<SequenceType>,
+ rchandle<exprnode>,
rchandle<GroupCollationSpec>);
const QName* get_var_name() const { return var_name_h.getp(); }
+ const rchandle<exprnode> get_var_expr() const {return var_init_expr_h;}
+
+ const rchandle<SequenceType> get_var_type() const {return var_type_h;}
+
rchandle<GroupCollationSpec> group_coll_spec() const { return group_coll_spec_h; }
void accept(parsenode_visitor&) const;
@@ -3106,7 +3114,7 @@
protected:
rchandle<exprnode> left;
rchandle<exprnode> right;
-
+
public:
StringConcatExpr(
const QueryLoc& aLoc,
=== modified file 'src/compiler/rewriter/rules/nodeid_rules.cpp'
--- src/compiler/rewriter/rules/nodeid_rules.cpp 2012-04-24 12:39:38 +0000
+++ src/compiler/rewriter/rules/nodeid_rules.cpp 2012-05-16 22:51:21 +0000
@@ -770,7 +770,8 @@
if (e->getSourceExpr() != NULL &&
(e->get_expr_kind() == insert_expr_kind ||
- static_cast<replace_expr*>(e)->getType() == store::UpdateConsts::NODE) &&
+ (e->get_expr_kind() == replace_expr_kind &&
+ static_cast<replace_expr*>(e)->getType() == store::UpdateConsts::NODE)) &&
(sctx->inherit_mode() != StaticContextConsts::no_inherit_ns ||
sctx->preserve_mode() != StaticContextConsts::no_preserve_ns))
{
=== modified file 'src/compiler/translator/translator.cpp'
--- src/compiler/translator/translator.cpp 2012-04-24 12:39:38 +0000
+++ src/compiler/translator/translator.cpp 2012-05-16 22:51:21 +0000
@@ -68,6 +68,8 @@
#include "functions/signature.h"
#include "functions/udf.h"
#include "functions/external_function.h"
+#include "functions/func_ft_module.h"
+#include "functions/func_ft_module_impl.h"
#include "annotations/annotations.h"
@@ -83,6 +85,7 @@
#include "zorbatypes/URI.h"
#include "zorbatypes/numconversions.h"
+#include "zorbamisc/ns_consts.h"
#ifdef ZORBA_WITH_DEBUGGER
#include "debugger/debugger_commons.h"
@@ -105,7 +108,6 @@
#define NODE_SORT_OPT
-#define XS_URI "http://www.w3.org/2001/XMLSchema";
namespace zorba
{
@@ -692,6 +694,10 @@
xquery_fns_def_dot.set(FunctionConsts::FN_DATA_0);
xquery_fns_def_dot.set(FunctionConsts::FN_DOCUMENT_URI_0);
xquery_fns_def_dot.set(FunctionConsts::FN_NODE_NAME_0);
+ xquery_fns_def_dot.set(FunctionConsts::FN_NILLED_0);
+ xquery_fns_def_dot.set(FunctionConsts::FN_HAS_CHILDREN_0);
+ xquery_fns_def_dot.set(FunctionConsts::FN_PATH_0);
+
op_concatenate = GET_BUILTIN_FUNCTION(OP_CONCATENATE_N);
assert(op_concatenate != NULL);
@@ -711,7 +717,7 @@
}
-~TranslatorImpl()
+~TranslatorImpl()
{
#ifndef ZORBA_NO_FULL_TEXT
while (!theFTNodeStack.empty())
@@ -859,7 +865,7 @@
{
ZORBA_ASSERT(count >= 0);
- ftnode *n = NULL;
+ ftnode *n = nullptr;
while ( count-- > 0 )
{
ZORBA_FATAL( !theFTNodeStack.empty(), "" );
@@ -1217,7 +1223,7 @@
method raises error.
If var is not found, the method raises the given error, unless the given error
- is MAX_ZORBA_ERROR_CODE, in which case it returns NULL.
+ is zerr::ZXQP0000_NO_ERROR, in which case it returns NULL.
********************************************************************************/
var_expr* lookup_var(
const QName* qname,
@@ -1774,7 +1780,7 @@
if (typeid(c) == typeid(ForClause))
{
- const VarInDeclList& varDecls =
+ const VarInDeclList& varDecls =
*(static_cast<const ForClause*>(&c)->get_vardecl_list());
for (int j = (int)varDecls.size() - 1; j >= 0; --j)
@@ -1789,7 +1795,7 @@
}
else if (typeid(c) == typeid(LetClause))
{
- const VarGetsDeclList& lV =
+ const VarGetsDeclList& lV =
*(static_cast<const LetClause*>(&c)->get_vardecl_list());
for (int j = (int)lV.size() - 1; j >= 0; --j)
@@ -1833,6 +1839,16 @@
}
else if (typeid (c) == typeid (GroupByClause))
{
+ const GroupByClause groupClause = *static_cast<const GroupByClause*>(&c);
+
+ //Group-by clauses may define new variables, otherwise it shadows existing vars.
+ for(size_t gSpecPos = 0; gSpecPos < groupClause.get_spec_list()->size(); gSpecPos++)
+ {
+ GroupSpec *groupSpec = (*(groupClause.get_spec_list()))[gSpecPos];
+ if(groupSpec->get_var_expr() != NULL)
+ vars.insert(lookup_var(groupSpec->get_var_name(), loc, err::XPST0008));
+ }
+
// Group-by redefines ALL previous variables, but the GroupByClause lists
// only the grouping vars. So, to find the var_exprs for the vars defined
// by the GroupByClause, we exploit the fact that the redefined var_exprs
@@ -2011,7 +2027,7 @@
theSctx->bind_ns(pfx, targetNS, loc, err::XQST0033);
}
- zstring xsdTNS = zstring(XS_URI);
+ zstring xsdTNS = zstring(XML_SCHEMA_NS);
if ( xsdTNS.compare(targetNS)==0 )
{
// Xerces doesn't like importing XMLSchema.xsd schema4schema, so we skip it
@@ -2279,7 +2295,7 @@
// If an appliaction set a type for the context item via the c++ api, then
// create a full declaration for it in order to enforce that type.
- if (!theHaveContextItemDecl &&
+ if (!theHaveContextItemDecl &&
theSctx->get_context_item_type() != theRTM.ITEM_TYPE_ONE.getp())
{
var_expr* var = lookup_ctx_var(DOT_VARNAME, loc);
@@ -2838,6 +2854,11 @@
// importing module that X has been imported.
if (atlist == NULL && static_context::is_builtin_module(targetNS))
{
+ // just a test, this will throw, if the access is denied
+ std::vector<zstring> candidateURIs;
+ theRootSctx->get_candidate_uris(targetNS,
+ internal::EntityData::MODULE,
+ candidateURIs);
theRootSctx->add_imported_builtin_module(targetNS);
#ifdef NDEBUG
// We cannot skip the math or the sctx introspection modules because they
@@ -3273,6 +3294,11 @@
if (f.getp() != 0)
{
+ if (f->isUdf())
+ {
+ RAISE_ERROR(err::XQST0034, loc, ERROR_PARAMS(qnameItem->getStringValue()));
+ }
+
// We make sure that the types of the parameters and the return type
// are subtypes of the ones declared in the module
const signature& s = f->getSignature();
@@ -3294,6 +3320,41 @@
qnameItem->getLocalName())));
}
+#ifndef ZORBA_NO_FULL_TEXT
+ if (qnameItem->getNamespace() == static_context::ZORBA_FULL_TEXT_FN_NS &&
+ (qnameItem->getLocalName() == "tokenizer-properties" ||
+ qnameItem->getLocalName() == "tokenize"))
+ {
+ FunctionConsts::FunctionKind kind;
+
+ if (qnameItem->getLocalName() == "tokenizer-properties")
+ {
+ assert(numParams <= 1);
+
+ if (numParams == 1)
+ kind = FunctionConsts::FULL_TEXT_TOKENIZER_PROPERTIES_1;
+ else
+ kind = FunctionConsts::FULL_TEXT_TOKENIZER_PROPERTIES_0;
+
+ f = new full_text_tokenizer_properties(f->getSignature(), kind);
+ }
+ else
+ {
+ assert(numParams == 1 || numParams == 2);
+
+ if (numParams == 2)
+ kind = FunctionConsts::FULL_TEXT_TOKENIZE_2;
+ else
+ kind = FunctionConsts::FULL_TEXT_TOKENIZE_1;
+
+ f = new full_text_tokenize(f->getSignature(), kind);
+ }
+
+ f->setStaticContext(theRootSctx);
+ bind_fn(f, numParams, loc);
+ }
+#endif /* ZORBA_NO_FULL_TEXT */
+
f->setAnnotations(theAnnotations);
theAnnotations = NULL; // important to reset
@@ -3609,7 +3670,7 @@
{
//lc->setLazyEval(!f->isSequential());
- const user_function* udf =
+ const user_function* udf =
static_cast<const user_function*>(theCurrentPrologVFDecl.getFunction());
arg_var->set_param_pos(flwor->num_clauses());
@@ -3779,7 +3840,7 @@
bind_var(ve, export_sctx);
#ifdef ZORBA_WITH_DEBUGGER
- if (initExpr != NULL && theCCB->theDebuggerCommons != NULL)
+ if (initExpr != NULL && theCCB->theDebuggerCommons != NULL)
{
QueryLoc lExpandedLocation = expandQueryLoc(v.get_name()->get_location(),
initExpr->get_loc());
@@ -6361,9 +6422,23 @@
for (int i = 0; i < (int)lList->size(); ++i)
{
GroupSpec* spec = (*lList)[i];
+
const QName* varname = spec->get_var_name();
- const var_expr* ve = lookup_var(varname, loc, err::XPST0008);
- group_vars.insert(ve);
+
+ const var_expr* ve = NULL;
+ if(spec->get_var_expr() == NULL)
+ {
+ ve = lookup_var(varname, loc, err::XPST0008);
+ }
+ else
+ {
+ //variables can be explicitly shadowed, if we don't check for that
+ //we might have them become non-group variables incorrectly.
+ ve = lookup_var(varname, loc, zerr::ZXQP0000_NO_ERROR);
+ }
+
+ if(ve != NULL)
+ group_vars.insert(ve);
}
// The non-grouping vars are the vars in the difference of the 2 sets above.
@@ -6382,9 +6457,7 @@
{
push_nodestack(const_cast<var_expr *>(*i));
- push_scope();
-
- var_expr_t ve = bind_var(loc, (*i)->get_name(), var_expr::non_groupby_var);
+ var_expr_t ve = create_var(loc, (*i)->get_name(), var_expr::non_groupby_var);
push_nodestack(ve.getp());
}
@@ -6396,6 +6469,7 @@
{
TRACE_VISIT_OUT();
+ // NOTE: Now grouping vars have a expr for the input stream.
// At this point, the nodestack contains a pair of var_exprs for each var X
// defined by any clauses appearing before this GroupByClause. The first
// var_expr in the pair corresponds to the input-stream var X, and the second
@@ -6409,28 +6483,40 @@
std::vector<std::string> collations;
group_clause::rebind_list_t grouping_rebind;
group_clause::rebind_list_t nongrouping_rebind;
+ expr_t input_expr;
var_expr_t input_var;
var_expr_t output_var;
+ std::vector<var_expr_t> output_vars_helper;
for (int i = (int)numGroupSpecs - 1; i >= 0; i--)
{
const GroupSpec& groupSpec = *groupSpecs[i];
output_var = pop_nodestack_var();
- input_var = pop_nodestack_var();
+ input_expr = pop_nodestack();
+
+ output_vars_helper.push_back(output_var);
if (groupSpec.group_coll_spec() != NULL)
collations.push_back(groupSpec.group_coll_spec()->get_uri().str());
else
collations.push_back ("");
- wrapper_expr_t input_wrapper;
- input_wrapper = new wrapper_expr(theRootSctx,
+ if(input_expr->get_expr_kind() == var_expr_kind)
+ input_expr = new wrapper_expr(theRootSctx,
loc,
- static_cast<expr*>(input_var.getp()));
-
- grouping_rebind.push_back(std::pair<wrapper_expr_t, var_expr_t>(input_wrapper,
- output_var));
+ input_expr.getp());
+
+ grouping_rebind.push_back(std::pair<expr_t, var_expr_t>(input_expr,
+ output_var));
+ }
+
+ for(std::vector<var_expr_t>::reverse_iterator iter = output_vars_helper.rbegin();
+ iter != output_vars_helper.rend();
+ iter++)
+ {
+ push_scope();
+ bind_var(*iter, theSctx);
}
reverse(collations.begin(), collations.end());
@@ -6440,6 +6526,9 @@
{
input_var = pop_nodestack_var();
+ push_scope();
+ bind_var(output_var, theSctx);
+
wrapper_expr_t input_wrapper;
input_wrapper = new wrapper_expr(theRootSctx,
loc,
@@ -6474,30 +6563,40 @@
/*******************************************************************************
- GroupSpec ::= "$" VarName ("collation" URILiteral)?
+ GroupSpec ::= "$" VarName (TypeDeclaration? ":=" ExprSingle)? ("collation" URILiteral)?
********************************************************************************/
void* begin_visit(const GroupSpec& v)
{
TRACE_VISIT();
-
- var_expr* e = lookup_var(v.get_var_name(), loc, err::XPST0008);
+ return no_state;
+}
+
+void end_visit(const GroupSpec& v, void* /*visit_state*/)
+{
+ TRACE_VISIT_OUT();
+
+ //wrap data
+ expr_t e = (v.get_var_expr() != NULL) ?
+ wrap_in_atomization(pop_nodestack()) : lookup_var(v.get_var_name(), loc, err::XPST0008);
+
+ //wrap type
+ xqtref_t type = NULL;
+ if(v.get_var_type() != NULL)
+ {
+ type = pop_tstack();
+ e = wrap_in_type_promotion(e, type);
+ }
// Create a new var_expr gX, corresponding to the input-stream var X that
// is referenced by this group spec. gX represents X in the output stream.
// Push the var_exprs for both X and gX into the nodestack.
- push_scope();
-
- push_nodestack(e);
-
- var_expr_t ve = bind_var(loc, v.get_var_name(), var_expr::groupby_var);
+
+ push_nodestack(e.getp());
+
+ var_expr_t ve =
+ create_var(loc, v.get_var_name(), var_expr::groupby_var, type);
+
push_nodestack(ve.getp());
-
- return no_state;
-}
-
-void end_visit(const GroupSpec& v, void* /*visit_state*/)
-{
- TRACE_VISIT_OUT();
}
@@ -9409,7 +9508,7 @@
expr_t right = pop_nodestack();
expr_t left = pop_nodestack();
concat_args.push_back(left);
-
+
//If the right leaf is the concat expr,
//we add directly its leafs to the new concat expr.
bool rightLeafIsConcatExpr = false;
@@ -9426,13 +9525,13 @@
}
}
}
-
+
if(!rightLeafIsConcatExpr)
{
concat_args.push_back(right);
}
- rchandle<expr> concat = new fo_expr(theRootSctx, loc, GET_BUILTIN_FUNCTION(FN_CONCAT_N), concat_args);
+ rchandle<expr> concat = new fo_expr(theRootSctx, loc, GET_BUILTIN_FUNCTION(FN_CONCAT_N), concat_args);
push_nodestack(concat);
}
@@ -12492,7 +12591,7 @@
#ifndef ZORBA_NO_FULL_TEXT
-template<typename FTNodeType> bool flatten( ftnode *n )
+template<typename FTNodeType> bool flatten( ftnode *n )
{
if ( FTNodeType *const n2 = dynamic_cast<FTNodeType*>( n ) ) {
typename FTNodeType::ftnode_list_t &list = n2->get_node_list();
@@ -12508,16 +12607,16 @@
}
#endif /* ZORBA_NO_FULL_TEXT */
-void *begin_visit (const FTAnd& v)
+void *begin_visit (const FTAnd& v)
{
TRACE_VISIT ();
#ifndef ZORBA_NO_FULL_TEXT
- push_ftstack( NULL ); // sentinel
+ push_ftstack( nullptr ); // sentinel
#endif /* ZORBA_NO_FULL_TEXT */
return no_state;
}
-void end_visit (const FTAnd& v, void* /*visit_state*/)
+void end_visit (const FTAnd& v, void* /*visit_state*/)
{
TRACE_VISIT_OUT ();
#ifndef ZORBA_NO_FULL_TEXT
@@ -12539,44 +12638,44 @@
}
-void *begin_visit (const FTAnyallOption& v)
-{
- TRACE_VISIT ();
- // nothing to do
- return no_state;
-}
-
-
-void end_visit (const FTAnyallOption& v, void* /*visit_state*/)
-{
- TRACE_VISIT_OUT ();
- // nothing to do
-}
-
-
-void *begin_visit (const FTBigUnit& v)
-{
- TRACE_VISIT ();
- // nothing to do
- return no_state;
-}
-
-
-void end_visit (const FTBigUnit& v, void* /*visit_state*/)
-{
- TRACE_VISIT_OUT ();
- // nothing to do
-}
-
-void *begin_visit (const FTCaseOption& v)
-{
- TRACE_VISIT ();
- // nothing to do
- return no_state;
-}
-
-
-void end_visit (const FTCaseOption& v, void* /*visit_state*/)
+void *begin_visit (const FTAnyallOption& v)
+{
+ TRACE_VISIT ();
+ // nothing to do
+ return no_state;
+}
+
+
+void end_visit (const FTAnyallOption& v, void* /*visit_state*/)
+{
+ TRACE_VISIT_OUT ();
+ // nothing to do
+}
+
+
+void *begin_visit (const FTBigUnit& v)
+{
+ TRACE_VISIT ();
+ // nothing to do
+ return no_state;
+}
+
+
+void end_visit (const FTBigUnit& v, void* /*visit_state*/)
+{
+ TRACE_VISIT_OUT ();
+ // nothing to do
+}
+
+void *begin_visit (const FTCaseOption& v)
+{
+ TRACE_VISIT ();
+ // nothing to do
+ return no_state;
+}
+
+
+void end_visit (const FTCaseOption& v, void* /*visit_state*/)
{
TRACE_VISIT_OUT ();
#ifndef ZORBA_NO_FULL_TEXT
@@ -12591,7 +12690,7 @@
}
-void *begin_visit (const FTContainsExpr& v)
+void *begin_visit (const FTContainsExpr& v)
{
TRACE_VISIT ();
#ifdef ZORBA_NO_FULL_TEXT
@@ -12602,7 +12701,7 @@
return no_state;
}
-void end_visit (const FTContainsExpr& v, void* /*visit_state*/)
+void end_visit (const FTContainsExpr& v, void* /*visit_state*/)
{
TRACE_VISIT_OUT ();
#ifndef ZORBA_NO_FULL_TEXT
@@ -12756,7 +12855,7 @@
void *begin_visit (const FTMildNot& v) {
TRACE_VISIT ();
#ifndef ZORBA_NO_FULL_TEXT
- push_ftstack( NULL ); // sentinel
+ push_ftstack( nullptr ); // sentinel
#endif /* ZORBA_NO_FULL_TEXT */
return no_state;
}
@@ -12799,7 +12898,7 @@
void *begin_visit (const FTOr& v) {
TRACE_VISIT ();
#ifndef ZORBA_NO_FULL_TEXT
- push_ftstack( NULL ); // sentinel
+ push_ftstack( nullptr ); // sentinel
#endif /* ZORBA_NO_FULL_TEXT */
return no_state;
}
@@ -13058,7 +13157,7 @@
levels = dynamic_cast<ftrange*>( pop_ftstack() );
ZORBA_ASSERT( levels );
} else
- levels = NULL;
+ levels = nullptr;
ftthesaurus_id *const tid = new ftthesaurus_id(
loc, v.get_uri(), v.get_relationship(), levels
@@ -13070,7 +13169,7 @@
void *begin_visit (const FTThesaurusOption& v) {
TRACE_VISIT ();
#ifndef ZORBA_NO_FULL_TEXT
- push_ftstack( NULL ); // sentinel
+ push_ftstack( nullptr ); // sentinel
#endif /* ZORBA_NO_FULL_TEXT */
return no_state;
}
@@ -13078,10 +13177,8 @@
void end_visit (const FTThesaurusOption& v, void* /*visit_state*/) {
TRACE_VISIT_OUT ();
#ifndef ZORBA_NO_FULL_TEXT
- ftthesaurus_id *default_tid = NULL;
- if ( v.includes_default() ) {
- default_tid = new ftthesaurus_id( loc, "##default" );
- }
+ ftthesaurus_id *const default_tid = v.includes_default() ?
+ new ftthesaurus_id( loc, "##default" ) : nullptr;
ftthesaurus_option::thesaurus_id_list_t list;
while ( true ) {
=== modified file 'src/context/CMakeLists.txt'
--- src/context/CMakeLists.txt 2012-04-24 12:39:38 +0000
+++ src/context/CMakeLists.txt 2012-05-16 22:51:21 +0000
@@ -32,11 +32,6 @@
features.cpp
)
-IF (NOT ZORBA_NO_FULL_TEXT)
- LIST(APPEND CONTEXT_SRCS
- thesaurus_wrappers.cpp)
-ENDIF (NOT ZORBA_NO_FULL_TEXT)
-
SET(CONTEXT_BUILD_SRCS
${CMAKE_CURRENT_BINARY_DIR}/context/root_static_context_init.cpp
)
=== modified file 'src/context/default_url_resolvers.cpp'
--- src/context/default_url_resolvers.cpp 2012-04-24 12:39:38 +0000
+++ src/context/default_url_resolvers.cpp 2012-05-16 22:51:21 +0000
@@ -17,6 +17,7 @@
#include "context/default_url_resolvers.h"
+#include "util/cxx_util.h"
#include "util/uri_util.h"
#include "util/http_util.h"
#include "util/fs_util.h"
@@ -41,8 +42,15 @@
HTTPURLResolver::resolveURL
(zstring const& aUrl, EntityData const* aEntityData)
{
- if (aEntityData->getKind() == EntityData::COLLECTION)
- return NULL;
+ switch ( aEntityData->getKind() ) {
+ case EntityData::COLLECTION:
+#ifndef ZORBA_NO_FULL_TEXT
+ case EntityData::THESAURUS:
+#endif /* ZORBA_NO_FULL_TEXT */
+ return nullptr;
+ default:
+ break;
+ }
uri::scheme lScheme = uri::get_scheme(aUrl);
switch (lScheme) {
@@ -82,8 +90,15 @@
FileURLResolver::resolveURL
(zstring const& aUrl, EntityData const* aEntityData)
{
- if (aEntityData->getKind() == EntityData::COLLECTION)
- return NULL;
+ switch ( aEntityData->getKind() ) {
+ case EntityData::COLLECTION:
+#ifndef ZORBA_NO_FULL_TEXT
+ case EntityData::THESAURUS:
+#endif /* ZORBA_NO_FULL_TEXT */
+ return nullptr;
+ default:
+ break;
+ }
uri::scheme lScheme = uri::get_scheme(aUrl);
if (lScheme != uri::file) {
@@ -111,7 +126,6 @@
{
if (aEntityData->getKind() != EntityData::COLLECTION)
return NULL;
-
store::Item_t lName;
GENV_STORE.getItemFactory()->createQName(lName, aUrl.c_str(), "", "zorba-internal-name-for-w3c-collections");
store::Collection_t lColl = GENV_STORE.getCollection(lName.getp(), true);
=== modified file 'src/context/static_context.cpp'
--- src/context/static_context.cpp 2012-04-24 12:39:38 +0000
+++ src/context/static_context.cpp 2012-05-16 22:51:21 +0000
@@ -378,11 +378,16 @@
static_context::ZORBA_XML_FN_NS =
"http://www.zorba-xquery.com/modules/xml";;
+#ifndef ZORBA_NO_FULL_TEXT
+const char*
+static_context::ZORBA_FULL_TEXT_FN_NS =
+"http://www.zorba-xquery.com/modules/full-text";;
+#endif /* ZORBA_NO_FULL_TEXT */
+
const char*
static_context::ZORBA_XML_FN_OPTIONS_NS =
"http://www.zorba-xquery.com/modules/xml-options";;
-
/***************************************************************************//**
Target namespaces of zorba reserved modules
********************************************************************************/
@@ -451,8 +456,11 @@
ns == ZORBA_JSON_FN_NS ||
ns == ZORBA_FETCH_FN_NS ||
ns == ZORBA_NODE_FN_NS ||
+#ifndef ZORBA_NO_FULL_TEXT
+ ns == ZORBA_FULL_TEXT_FN_NS ||
+#endif /* ZORBA_NO_FULL_TEXT */
ns == ZORBA_XML_FN_NS);
- }
+ }
else if (ns == W3C_FN_NS || ns == XQUERY_MATH_FN_NS)
{
return true;
@@ -488,6 +496,11 @@
Static method to check if a given target namespace identifies a zorba non
pure builtin module, i.e. a builtin module that, in addition to builtin
external functions, contains variable declarations and/or udfs.
+
+ Note: The fuul-text module must be included here because it MUST be processed
+ when imported, even in RELEASE mode. The reason is that the tokenize and
+ tokenizer-properties functions must be registered in the module's sctx (in
+ addition to the root sctx).
********************************************************************************/
bool static_context::is_non_pure_builtin_module(const zstring& ns)
{
@@ -499,6 +512,9 @@
ns == ZORBA_JSON_FN_NS ||
ns == ZORBA_URI_FN_NS ||
ns == ZORBA_RANDOM_FN_NS ||
+#ifndef ZORBA_NO_FULL_TEXT
+ ns == ZORBA_FULL_TEXT_FN_NS ||
+#endif /* ZORBA_NO_FULL_TEXT */
ns == ZORBA_XML_FN_NS);
}
@@ -1510,6 +1526,21 @@
}
}
+void static_context::get_candidate_uris(
+ zstring const& aUri,
+ internal::EntityData::Kind aEntityKind,
+ std::vector<zstring>& oComponents) const
+{
+ // Create a simple EntityData that just reports the specified Kind
+ internal::EntityData const lData(aEntityKind);
+
+ apply_uri_mappers(aUri, &lData, internal::URIMapper::CANDIDATE, oComponents);
+ if (oComponents.size() == 0)
+ {
+ oComponents.push_back(aUri);
+ }
+}
+
/***************************************************************************//**
@@ -1585,7 +1616,7 @@
std::auto_ptr<internal::Resource>& oResource,
zstring& oErrorMessage) const
{
- oErrorMessage = "";
+ oErrorMessage.clear();
// Iterate through all candidate URLs...
for (std::vector<zstring>::iterator url = aUrls.begin();
@@ -1621,7 +1652,7 @@
}
catch (const std::exception& e)
{
- if (oErrorMessage == "")
+ if (oErrorMessage.empty())
{
// Really no point in saving anything more than the first message
oErrorMessage = e.what();
@@ -2277,9 +2308,7 @@
if (!is_global_root_sctx() && lookup_local_fn(qname, arity) != NULL)
{
- throw XQUERY_EXCEPTION(
- err::XQST0034, ERROR_PARAMS( qname->getStringValue() ), ERROR_LOC( loc )
- );
+ RAISE_ERROR(err::XQST0034, loc, ERROR_PARAMS(qname->getStringValue()));
}
if (theFunctionMap == NULL)
=== modified file 'src/context/static_context.h'
--- src/context/static_context.h 2012-04-24 12:39:38 +0000
+++ src/context/static_context.h 2012-05-16 22:51:21 +0000
@@ -471,6 +471,9 @@
static const char* ZORBA_FETCH_FN_NS;
static const char* ZORBA_NODE_FN_NS;
static const char* ZORBA_XML_FN_NS;
+#ifndef ZORBA_NO_FULL_TEXT
+ static const char* ZORBA_FULL_TEXT_FN_NS;
+#endif /* ZORBA_NO_FULL_TEXT */
static const char* ZORBA_XML_FN_OPTIONS_NS;
// Namespaces of virtual modules declaring zorba builtin functions
@@ -702,6 +705,15 @@
(zstring const& aUri, internal::EntityData::Kind aEntityKind,
std::vector<zstring>& oComponents) const;
+ /**
+ * Given a URI, populate a vector with a list of candidate URIs. If
+ * no candidate URIs are available, the vector will be populated
+ * with (only) the input URI.
+ */
+ void get_candidate_uris
+ (zstring const& aUri, internal::EntityData::Kind aEntityKind,
+ std::vector<zstring>& oComponents) const;
+
void set_uri_path(const std::vector<zstring>& aURIPath);
void get_uri_path(std::vector<zstring>& oURIPath) const;
=== removed file 'src/context/stemmer_wrappers.cpp'
--- src/context/stemmer_wrappers.cpp 2012-04-24 12:39:38 +0000
+++ src/context/stemmer_wrappers.cpp 1970-01-01 00:00:00 +0000
@@ -1,74 +0,0 @@
-/*
- * Copyright 2006-2008 The FLWOR Foundation.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-#include "stdafx.h"
-
-#include <zorba/config.h>
-
-#ifndef ZORBA_NO_FULL_TEXT
-
-#include "api/unmarshaller.h"
-#include "diagnostics/assert.h"
-#include "util/cxx_util.h"
-
-#include "stemmer_wrappers.h"
-
-using namespace zorba::locale;
-
-namespace zorba {
-namespace internal {
-
-///////////////////////////////////////////////////////////////////////////////
-
-StemmerWrapper::StemmerWrapper( zorba::Stemmer const *s ) :
- api_stemmer_( s )
-{
- ZORBA_ASSERT( api_stemmer_ );
-}
-
-void StemmerWrapper::stem( zstring const &word, iso639_1::type lang,
- zstring *result ) const {
- String const api_word( Unmarshaller::newString( word ) );
- String api_result( Unmarshaller::newString( *result ) );
- api_stemmer_->stem( api_word, lang, &api_result );
-}
-
-///////////////////////////////////////////////////////////////////////////////
-
-StemmerProviderWrapper::
-StemmerProviderWrapper( zorba::StemmerProvider const *p ) :
- api_stemmer_provider_( p )
-{
- ZORBA_ASSERT( api_stemmer_provider_ );
-}
-
-Stemmer const*
-StemmerProviderWrapper::get_stemmer( iso639_1::type lang ) const {
- zorba::Stemmer const *const s = api_stemmer_provider_->getStemmer( lang );
- return s ? new StemmerWrapper( s ) : nullptr;
-}
-
-///////////////////////////////////////////////////////////////////////////////
-
-} // namespace internal
-} // namespace zorba
-
-#endif /* ZORBA_NO_FULL_TEXT */
-/*
- * Local variables:
- * mode: c++
- * End:
- */
-/* vim:set et sw=2 ts=2: */
=== removed file 'src/context/stemmer_wrappers.h'
--- src/context/stemmer_wrappers.h 2012-04-24 12:39:38 +0000
+++ src/context/stemmer_wrappers.h 1970-01-01 00:00:00 +0000
@@ -1,63 +0,0 @@
-/*
- * Copyright 2006-2008 The FLWOR Foundation.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-#pragma once
-#ifndef ZORBA_STEMMER_WRAPPERS_H
-#define ZORBA_STEMMER_WRAPPERS_H
-
-#include <zorba/config.h>
-
-#if 0
-#ifndef ZORBA_NO_FULL_TEXT
-
-#include <zorba/stemmer.h>
-#include "zorbautils/stemmer.h"
-
-namespace zorba {
-namespace internal {
-
-///////////////////////////////////////////////////////////////////////////////
-
-class StemmerWrapper : public Stemmer {
-public:
- StemmerWrapper( zorba::Stemmer const *api_stemmer );
- void stem( zstring const &word, locale::iso639_1::type lang,
- zstring *result ) const;
-private:
- zorba::Stemmer const *const api_stemmer_;
-};
-
-class StemmerProviderWrapper : public StemmerProvider {
-public:
- StemmerProviderWrapper( zorba::StemmerProvider const *p );
- Stemmer const* get_stemmer( locale::iso639_1::type lang ) const;
-private:
- zorba::StemmerProvider const *const api_stemmer_provider_;
-};
-
-///////////////////////////////////////////////////////////////////////////////
-
-} // namespace internal
-} // namespace zorba
-
-#endif /* ZORBA_NO_FULL_TEXT */
-#endif
-#endif /* ZORBA_STEMMER_WRAPPERS_H */
-/*
- * Local variables:
- * mode: c++
- * End:
- */
-/* vim:set et sw=2 ts=2: */
=== modified file 'src/context/uri_resolver.cpp'
--- src/context/uri_resolver.cpp 2012-04-24 12:39:38 +0000
+++ src/context/uri_resolver.cpp 2012-05-16 22:51:21 +0000
@@ -117,19 +117,6 @@
{
}
-#ifndef ZORBA_NO_FULL_TEXT
- ThesaurusEntityData::ThesaurusEntityData(locale::iso639_1::type aLang)
- : EntityData(EntityData::THESAURUS),
- theLang(aLang)
- {
- }
-
- locale::iso639_1::type ThesaurusEntityData::getLanguage() const
- {
- return theLang;
- }
-#endif /* ZORBA_NO_FULL_TEXT */
-
/*************
* URIMapper is an abstract class, but we have to define its vtbl and
* base destructor somewhere.
=== modified file 'src/context/uri_resolver.h'
--- src/context/uri_resolver.h 2012-04-24 12:39:38 +0000
+++ src/context/uri_resolver.h 2012-05-16 22:51:21 +0000
@@ -55,21 +55,21 @@
/**
* @brief Return the URL used to load this Resource.
*/
- zstring getUrl() { return theUrl; }
+ zstring const& getUrl() const { return theUrl; }
virtual ~Resource() = 0;
- protected:
+protected:
Resource();
- private:
+private:
/**
* Used by static_context to populate the URL.
*/
+ void setUrl(zstring const &aUrl) { theUrl = aUrl; }
friend class zorba::static_context;
- void setUrl(zstring aUrl) { theUrl = aUrl; }
zstring theUrl;
};
@@ -193,25 +193,6 @@
Kind const theKind;
};
-#ifndef ZORBA_NO_FULL_TEXT
-/**
- * @brief The class containing additional data for URIMappers and URLResolvers
- * when mapping/resolving a Thesaurus URI.
- */
-class ThesaurusEntityData : public EntityData
-{
-public:
- ThesaurusEntityData(locale::iso639_1::type aLang);
- /**
- * @brief Return the language for which a thesaurus is being requested.
- */
- virtual locale::iso639_1::type getLanguage() const;
-
-private:
- locale::iso639_1::type const theLang;
-};
-#endif /* ZORBA_NO_FULL_TEXT */
-
/**
* @brief Interface for URL resolving.
*
=== modified file 'src/diagnostics/assert.cpp'
--- src/diagnostics/assert.cpp 2012-04-24 12:39:38 +0000
+++ src/diagnostics/assert.cpp 2012-05-16 22:51:21 +0000
@@ -68,7 +68,7 @@
file,
line,
zerr::ZXQP0002_ASSERT_FAILED,
- ( msg ? ERROR_PARAMS( condition, msg ) : ERROR_PARAMS( condition ))
+ ( msg ? ERROR_PARAMS( condition, msg ) : ERROR_PARAMS( condition ) )
);
}
=== modified file 'src/diagnostics/assert.h'
--- src/diagnostics/assert.h 2012-04-24 12:39:38 +0000
+++ src/diagnostics/assert.h 2012-05-16 22:51:21 +0000
@@ -20,6 +20,10 @@
#ifndef ZORBA_ASSERT_H
#define ZORBA_ASSERT_H
+#include <sstream>
+
+#include "util/cxx_util.h"
+
namespace zorba {
/**
@@ -35,7 +39,7 @@
void assertion_failed( char const *condition,
char const *file,
int line,
- char const *msg = 0);
+ char const *msg = nullptr );
/**
* Zorba version of the standard assert(3) macro.
=== modified file 'src/diagnostics/diagnostic_en.xml'
--- src/diagnostics/diagnostic_en.xml 2012-04-24 12:39:38 +0000
+++ src/diagnostics/diagnostic_en.xml 2012-05-16 22:51:21 +0000
@@ -791,7 +791,19 @@
raise this error and instead provide some other implementation-defined
behavior.
</comment>
- <value>"$1": unsupported language</value>
+ <value>"$1": unsupported language${ 2}</value>
+ <entry key="BadStopWordsLang">
+ <value>for current stop words</value>
+ </entry>
+ <entry key="BadStemmerLang">
+ <value>for current stemmer</value>
+ </entry>
+ <entry key="BadThesaurusLang">
+ <value>for given thesaurus</value>
+ </entry>
+ <entry key="BadTokenizerLang">
+ <value>for current tokenizer</value>
+ </entry>
</diagnostic>
<diagnostic code="FTDY0016" if="!defined(ZORBA_NO_FULL_TEXT)">
@@ -983,7 +995,7 @@
<comment>
Invalid \c fn:format-number() picture string.
</comment>
- <value>"$1": invalid fn:format-number() picture string</value>
+ <value>"$1": invalid fn:format-number() picture string$2</value>
</diagnostic>
<diagnostic code="FODT0001">
@@ -1746,7 +1758,7 @@
<diagnostic code="ZXQP8401" name="THESAURUS_VERSION_MISMATCH"
if="!defined(ZORBA_NO_FULL_TEXT)">
<comment>
- The version of the thesaurus is not the expected version.
+ The version of the thesaurus is not the expected version.
</comment>
<value>"$1": wrong WordNet file version; should be "$2"</value>
</diagnostic>
@@ -1754,15 +1766,15 @@
<diagnostic code="ZXQP8402" name="THESAURUS_ENDIANNESS_MISMATCH"
if="!defined(ZORBA_NO_FULL_TEXT)">
<comment>
+ The thesaurus data file's endianness does not match that of the CPU.
</comment>
<value>thesaurus data endianness does not match CPU</value>
- The thesaurus data file's endianness does not match that of the CPU.
</diagnostic>
<diagnostic code="ZXQP8403" name="THESAURUS_DATA_ERROR"
if="!defined(ZORBA_NO_FULL_TEXT)">
<comment>
- The thesaurus data contains an unexpected value.
+ The thesaurus data contains an unexpected value.
</comment>
<value>thesaurus data error${: 1}</value>
</diagnostic>
@@ -3643,6 +3655,31 @@
<entry key="ParseFragmentInvalidOptions">
<value>invalid options passed to the parse-xml:parse() function, the element must in the schema target namespace</value>
</entry>
+
+ <entry key="FormatNumberDuplicates">
+ <value>: a sub-picture must not contain more than one of the "$3" sign</value>
+ </entry>
+
+ <entry key="FormatNumberGroupingAdjacentToDecimal">
+ <value>: a sub-picture must not contain a grouping-separator-sign adjacent to a decimal-separator-sign</value>
+ </entry>
+
+ <entry key="FormatNumberIntegerPart">
+ <value>: the integer part of a sub-picture must not contain a member of the decimal-digit-family that is followed by an optional-digit-sign</value>
+ </entry>
+
+ <entry key="FormatNumberFractionalPart">
+ <value>: the fractional part of a sub-picture must not contain an optional-digit-sign that is followed by a member of the decimal-digit-family</value>
+ </entry>
+
+ <entry key="FormatNumberPercentPermille">
+ <value>: a sub-picture must not contain more than one percent-sign or per-mille-sign, and it must not contain one of each</value>
+ </entry>
+
+ <entry key="FormatNumberAtLeastOneOptionalOrDecimal">
+ <value>: a sub-picture must contain at least one character that is an optional-digit-sign or a member of the decimal-digit-family</value>
+ </entry>
+
</subvalues>
=== modified file 'src/diagnostics/pregenerated/dict_en.cpp'
--- src/diagnostics/pregenerated/dict_en.cpp 2012-04-24 12:39:38 +0000
+++ src/diagnostics/pregenerated/dict_en.cpp 2012-05-16 22:51:21 +0000
@@ -47,7 +47,7 @@
{ "FODC0006", "invalid content passed to $1: $2" },
{ "FODC0007", "\"$1\": base URI passed to fn:parse() is not a valid absolute URI" },
{ "FODF1280", "\"$1\": invalid decimal format name for fn:format-number()" },
- { "FODF1310", "\"$1\": invalid fn:format-number() picture string" },
+ { "FODF1310", "\"$1\": invalid fn:format-number() picture string$2" },
{ "FODT0001", "overflow/underflow in date/time operation" },
{ "FODT0002", "overflow/underflow in duration operation" },
{ "FODT0003", "\"$1\": invalid timezone value" },
@@ -89,7 +89,7 @@
{ "FTST0008", "\"$1\": unknown stop-word list" },
#endif
#if !defined(ZORBA_NO_FULL_TEXT)
- { "FTST0009", "\"$1\": unsupported language" },
+ { "FTST0009", "\"$1\": unsupported language${ 2}" },
#endif
#if !defined(ZORBA_NO_FULL_TEXT)
{ "FTST0018", "\"$1\": unknown thesaurus" },
@@ -511,9 +511,19 @@
{ "~ExprReturnsTooManyUpdateLists", "expression does not return a pending update list" },
{ "~ExternFnDeterministic", "only external functions may be declared deterministic" },
{ "~ExternFnNondeterministic", "only external functions may be declared nondeterministic" },
+ { "~FTST0009_BadStemmerLang", "for current stemmer" },
+ { "~FTST0009_BadStopWordsLang", "for current stop words" },
+ { "~FTST0009_BadThesaurusLang", "for given thesaurus" },
+ { "~FTST0009_BadTokenizerLang", "for current tokenizer" },
{ "~FileNotFoundOrReadable", "file not found or readable" },
{ "~FnNilledArgNotNode", "fn:nilled() argument not a node" },
{ "~FnOnlyInXQueryVersion_3", "function only available in XQuery $3" },
+ { "~FormatNumberAtLeastOneOptionalOrDecimal", ": a sub-picture must contain at least one character that is an optional-digit-sign or a member of the decimal-digit-family" },
+ { "~FormatNumberDuplicates", ": a sub-picture must not contain more than one of the \"$3\" sign" },
+ { "~FormatNumberFractionalPart", ": the fractional part of a sub-picture must not contain an optional-digit-sign that is followed by a member of the decimal-digit-family" },
+ { "~FormatNumberGroupingAdjacentToDecimal", ": a sub-picture must not contain a grouping-separator-sign adjacent to a decimal-separator-sign" },
+ { "~FormatNumberIntegerPart", ": the integer part of a sub-picture must not contain a member of the decimal-digit-family that is followed by an optional-digit-sign" },
+ { "~FormatNumberPercentPermille", ": a sub-picture must not contain more than one percent-sign or per-mille-sign, and it must not contain one of each" },
{ "~FullTextNotEnabled", "full-text was not enabled in this build" },
{ "~FunctionFailedErrorCodeMessage_123", "$2 failed (error $3): $4" },
{ "~FunctionFailed_12o", "$2 failed${: 3}" },
=== modified file 'src/functions/CMakeLists.txt'
--- src/functions/CMakeLists.txt 2012-04-24 12:39:38 +0000
+++ src/functions/CMakeLists.txt 2012-05-16 22:51:21 +0000
@@ -83,3 +83,7 @@
func_apply.cpp
func_serialize_impl.cpp
)
+
+IF (NOT ZORBA_NO_FULL_TEXT)
+ LIST(APPEND FUNCTIONS_SRCS func_ft_module_impl.cpp)
+ENDIF (NOT ZORBA_NO_FULL_TEXT)
=== modified file 'src/functions/external_function.cpp'
--- src/functions/external_function.cpp 2012-04-24 12:39:38 +0000
+++ src/functions/external_function.cpp 2012-05-16 22:51:21 +0000
@@ -45,12 +45,12 @@
:
function(sig, FunctionConsts::FN_UNKNOWN),
theLoc(loc),
- theModuleSctx(modSctx),
theNamespace(ns),
theScriptingKind(scriptingType),
theImpl(impl)
{
resetFlag(FunctionConsts::isBuiltin);
+ theModuleSctx = modSctx;
}
@@ -62,7 +62,6 @@
zorba::serialization::serialize_baseclass(ar, (function*)this);
ar & theLoc;
- ar & theModuleSctx;
ar & theNamespace;
ar & theScriptingKind;
=== modified file 'src/functions/external_function.h'
--- src/functions/external_function.h 2012-04-24 12:39:38 +0000
+++ src/functions/external_function.h 2012-05-16 22:51:21 +0000
@@ -29,7 +29,6 @@
/*******************************************************************************
theLoc : The location of the declaration of this external function.
- theModuleContext : The root sctx of the module containing the declaration.
theNamespace : The namespace of the module containing the declaration.
theScriptingKind : Whether the external function is simple, updating, or
sequential (this property is part of the declaration).
@@ -40,7 +39,6 @@
{
protected:
QueryLoc theLoc;
- static_context * theModuleSctx;
zstring theNamespace;
short theScriptingKind;
ExternalFunction * theImpl;
=== added file 'src/functions/func_ft_module_impl.cpp'
--- src/functions/func_ft_module_impl.cpp 1970-01-01 00:00:00 +0000
+++ src/functions/func_ft_module_impl.cpp 2012-05-16 22:51:21 +0000
@@ -0,0 +1,128 @@
+/*
+ * Copyright 2006-2008 The FLWOR Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "functions/func_ft_module_impl.h"
+
+#include "runtime/full_text/ft_module.h"
+
+#define FT_MODULE_NS "http://www.zorba-xquery.com/modules/full-text";
+
+namespace zorba
+{
+
+#ifndef ZORBA_NO_FULL_TEXT
+
+SERIALIZABLE_CLASS_VERSIONS(full_text_tokenize)
+
+
+void full_text_tokenize::serialize(::zorba::serialization::Archiver& ar)
+{
+ serialize_baseclass(ar, (function*)this);
+}
+
+
+PlanIter_t full_text_tokenize::codegen(
+ CompilerCB*,
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& argv,
+ expr& ann) const
+{
+ return new TokenizeIterator(theModuleSctx, loc, argv);
+}
+
+
+SERIALIZABLE_CLASS_VERSIONS(full_text_tokenizer_properties)
+
+
+void full_text_tokenizer_properties::serialize(::zorba::serialization::Archiver& ar)
+{
+ serialize_baseclass(ar, (function*)this);
+}
+
+
+PlanIter_t full_text_tokenizer_properties::codegen(
+ CompilerCB*,
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& argv,
+ expr& ann) const
+{
+ return new TokenizerPropertiesIterator(theModuleSctx, loc, argv);
+}
+
+#endif // ZORBA_NO_FULL_TEXT
+
+
+///////////////////////////////////////////////////////////////////////////////
+
+void populate_context_ft_module_impl(static_context* sctx)
+{
+#ifndef ZORBA_NO_FULL_TEXT
+
+ xqtref_t tokenize_return_type =
+ GENV_TYPESYSTEM.create_node_type(store::StoreConsts::elementNode,
+ createQName(FT_MODULE_NS, "", "token"),
+ NULL,
+ TypeConstants::QUANT_STAR,
+ false,
+ false);
+ {
+ DECL_WITH_KIND(sctx,
+ full_text_tokenize,
+ (createQName(FT_MODULE_NS, "", "tokenize"),
+ GENV_TYPESYSTEM.ANY_NODE_TYPE_ONE,
+ tokenize_return_type),
+ FunctionConsts::FULL_TEXT_TOKENIZE_1);
+ }
+ {
+ DECL_WITH_KIND(sctx,
+ full_text_tokenize,
+ (createQName( FT_MODULE_NS, "", "tokenize"),
+ GENV_TYPESYSTEM.ANY_NODE_TYPE_ONE,
+ GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
+ tokenize_return_type),
+ FunctionConsts::FULL_TEXT_TOKENIZE_2);
+ }
+
+ xqtref_t tokenizer_properties_return_type =
+ GENV_TYPESYSTEM.create_node_type(store::StoreConsts::elementNode,
+ createQName(FT_MODULE_NS, "", "tokenizer-properties"),
+ NULL,
+ TypeConstants::QUANT_ONE,
+ false,
+ false);
+ {
+ DECL_WITH_KIND(sctx,
+ full_text_tokenizer_properties,
+ (createQName(FT_MODULE_NS, "", "tokenizer-properties"),
+ tokenizer_properties_return_type),
+ FunctionConsts::FULL_TEXT_TOKENIZER_PROPERTIES_0);
+ }
+ {
+ DECL_WITH_KIND(sctx,
+ full_text_tokenizer_properties,
+ (createQName( FT_MODULE_NS, "", "tokenizer-properties"),
+ GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
+ tokenizer_properties_return_type),
+ FunctionConsts::FULL_TEXT_TOKENIZER_PROPERTIES_1);
+ }
+#endif // ZORBA_NO_FULL_TEXT
+}
+
+
+
+} // namespace zorba
+/* vim:set et sw=2 ts=2: */
=== added file 'src/functions/func_ft_module_impl.h'
--- src/functions/func_ft_module_impl.h 1970-01-01 00:00:00 +0000
+++ src/functions/func_ft_module_impl.h 2012-05-16 22:51:21 +0000
@@ -0,0 +1,81 @@
+/*
+ * Copyright 2006-2008 The FLWOR Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef ZORBA_FUNCTIONS_FT_MODULE_IMPL_H
+#define ZORBA_FUNCTIONS_FT_MODULE_IMPL_H
+
+#include "stdafx.h"
+
+#include "functions/function.h"
+#include "functions/function_impl.h"
+
+
+namespace zorba
+{
+
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:tokenize
+class full_text_tokenize : public function
+{
+public:
+ SERIALIZABLE_CLASS(full_text_tokenize);
+ SERIALIZABLE_CLASS_CONSTRUCTOR2(full_text_tokenize, function)
+ void serialize(::zorba::serialization::Archiver& ar);
+
+public:
+ full_text_tokenize(const signature& sig, FunctionConsts::FunctionKind kind)
+ :
+ function(sig, kind)
+ {
+
+ }
+
+ CODEGEN_DECL();
+};
+
+
+
+//full-text:tokenizer-properties
+class full_text_tokenizer_properties : public function
+{
+public:
+ SERIALIZABLE_CLASS(full_text_tokenizer_properties);
+ SERIALIZABLE_CLASS_CONSTRUCTOR2(full_text_tokenizer_properties, function)
+ void serialize(::zorba::serialization::Archiver& ar);
+
+public:
+ full_text_tokenizer_properties(const signature& sig, FunctionConsts::FunctionKind kind)
+ :
+ function(sig, kind)
+ {
+
+ }
+
+ // Mark the function as accessing the dyn ctx so that it won't be
+ // const-folded. We must prevent const-folding because the function
+ // returns a node that is validated with a schema that may not be
+ // imported in the module where the function is invoked from.
+ bool accessesDynCtx() const { return true; }
+
+ CODEGEN_DECL();
+};
+
+#endif // ZORBA_NO_FULL_TEXT
+
+}
+
+#endif
+/* vim:set et sw=2 ts=2: */
=== modified file 'src/functions/function.cpp'
--- src/functions/function.cpp 2012-04-24 12:39:38 +0000
+++ src/functions/function.cpp 2012-05-16 22:51:21 +0000
@@ -43,6 +43,7 @@
theSignature(sig),
theKind(kind),
theFlags(0),
+ theModuleSctx(NULL),
theXQueryVersion(StaticContextConsts::xquery_version_1_0)
{
setFlag(FunctionConsts::isBuiltin);
@@ -70,6 +71,7 @@
SERIALIZE_ENUM(FunctionConsts::FunctionKind, theKind);
ar & theFlags;
ar & theAnnotationList;
+ ar & theModuleSctx;
SERIALIZE_ENUM(StaticContextConsts::xquery_version_t, theXQueryVersion);
}
@@ -92,6 +94,7 @@
return n == VARIADIC_SIG_SIZE || argv.size() == n;
}
+
/*******************************************************************************
********************************************************************************/
=== modified file 'src/functions/function.h'
--- src/functions/function.h 2012-04-24 12:39:38 +0000
+++ src/functions/function.h 2012-05-16 22:51:21 +0000
@@ -42,7 +42,10 @@
/*******************************************************************************
-
+ theModuleContext:
+ -----------------
+ The root sctx of the module containing the declaration. It is NULL for
+ functions that must be executed in the static context of the caller.
********************************************************************************/
class function : public SimpleRCObject
{
@@ -51,6 +54,7 @@
FunctionConsts::FunctionKind theKind;
uint32_t theFlags;
AnnotationList_t theAnnotationList;
+ static_context * theModuleSctx;
StaticContextConsts::xquery_version_t theXQueryVersion;
@@ -89,6 +93,10 @@
bool isVariadic() const { return theSignature.isVariadic(); }
+ static_context* getStaticContext() const { return theModuleSctx; }
+
+ void setStaticContext(static_context* sctx) { theModuleSctx = sctx; }
+
void setFlag(FunctionConsts::AnnotationFlags flag)
{
theFlags |= flag;
=== modified file 'src/functions/function_consts.h'
--- src/functions/function_consts.h 2012-04-24 12:39:38 +0000
+++ src/functions/function_consts.h 2012-05-16 22:51:21 +0000
@@ -225,6 +225,13 @@
OP_HOIST_1,
OP_UNHOIST_1,
+#ifndef ZORBA_NO_FULL_TEXT
+ FULL_TEXT_TOKENIZER_PROPERTIES_1,
+ FULL_TEXT_TOKENIZER_PROPERTIES_0,
+ FULL_TEXT_TOKENIZE_2,
+ FULL_TEXT_TOKENIZE_1,
+#endif
+
#include "functions/function_enum.h"
FN_MAX_FUNC
=== modified file 'src/functions/library.cpp'
--- src/functions/library.cpp 2012-04-24 12:39:38 +0000
+++ src/functions/library.cpp 2012-05-16 22:51:21 +0000
@@ -68,6 +68,10 @@
#include "functions/func_reflection.h"
#include "functions/func_apply.h"
#include "functions/func_fetch.h"
+#ifndef ZORBA_NO_FULL_TEXT
+#include "functions/func_ft_module.h"
+#include "runtime/full_text/ft_module_impl.h"
+#endif /* ZORBA_NO_FULL_TEXT */
#include "functions/func_function_item_iter.h"
@@ -144,6 +148,10 @@
populate_context_apply(sctx);
populate_context_fetch(sctx);
+#ifndef ZORBA_NO_FULL_TEXT
+ populate_context_ft_module(sctx);
+ populate_context_ft_module_impl(sctx);
+#endif /* ZORBA_NO_FULL_TEXT */
ar.set_loading_hardcoded_objects(false);
}
=== modified file 'src/functions/pregenerated/func_accessors.cpp'
--- src/functions/pregenerated/func_accessors.cpp 2012-04-24 12:39:38 +0000
+++ src/functions/pregenerated/func_accessors.cpp 2012-05-16 22:51:21 +0000
@@ -50,6 +50,15 @@
return new NodeNameIterator(sctx, loc, argv);
}
+PlanIter_t fn_nilled_3_0::codegen(
+ CompilerCB*,
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& argv,
+ expr& ann) const
+{
+ return new NilledIterator(sctx, loc, argv);
+}
PlanIter_t fn_nilled::codegen(
CompilerCB*,
static_context* sctx,
@@ -138,6 +147,17 @@
{
+ DECL_WITH_KIND(sctx, fn_nilled_3_0,
+ (createQName("http://www.w3.org/2005/xpath-functions","","nilled";),
+ GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+ FunctionConsts::FN_NILLED_0);
+
+ }
+
+
+ {
+
+
DECL_WITH_KIND(sctx, fn_nilled,
(createQName("http://www.w3.org/2005/xpath-functions","","nilled";),
GENV_TYPESYSTEM.ANY_NODE_TYPE_QUESTION,
=== modified file 'src/functions/pregenerated/func_accessors.h'
--- src/functions/pregenerated/func_accessors.h 2012-04-24 12:39:38 +0000
+++ src/functions/pregenerated/func_accessors.h 2012-05-16 22:51:21 +0000
@@ -72,6 +72,22 @@
//fn:nilled
+class fn_nilled_3_0 : public function
+{
+public:
+ fn_nilled_3_0(const signature& sig, FunctionConsts::FunctionKind kind)
+ :
+ function(sig, kind)
+ {
+theXQueryVersion = StaticContextConsts::xquery_version_3_0;
+ }
+
+ bool mustCopyInputNodes(expr* fo, csize producer) const { return true; }
+
+ CODEGEN_DECL();
+};
+
+//fn:nilled
class fn_nilled : public function
{
public:
@@ -82,7 +98,7 @@
}
- bool mustCopyInputNodes(expr* fo, csize producer) const { return false; }
+ bool mustCopyInputNodes(expr* fo, csize producer) const { return true; }
CODEGEN_DECL();
};
=== added file 'src/functions/pregenerated/func_ft_module.cpp'
--- src/functions/pregenerated/func_ft_module.cpp 1970-01-01 00:00:00 +0000
+++ src/functions/pregenerated/func_ft_module.cpp 2012-05-16 22:51:21 +0000
@@ -0,0 +1,490 @@
+/*
+ * Copyright 2006-2008 The FLWOR Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// ******************************************
+// * *
+// * THIS IS A GENERATED FILE. DO NOT EDIT! *
+// * SEE .xml FILE WITH SAME NAME *
+// * *
+// ******************************************
+
+
+#include "stdafx.h"
+#include "runtime/full_text/ft_module.h"
+#include "functions/func_ft_module.h"
+
+
+namespace zorba{
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_current_lang::codegen(
+ CompilerCB*,
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& argv,
+ expr& ann) const
+{
+ return new CurrentLangIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_host_lang::codegen(
+ CompilerCB*,
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& argv,
+ expr& ann) const
+{
+ return new HostLangIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_is_stem_lang_supported::codegen(
+ CompilerCB*,
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& argv,
+ expr& ann) const
+{
+ return new IsStemLangSupportedIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_is_stop_word::codegen(
+ CompilerCB*,
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& argv,
+ expr& ann) const
+{
+ return new IsStopWordIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_is_stop_word_lang_supported::codegen(
+ CompilerCB*,
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& argv,
+ expr& ann) const
+{
+ return new IsStopWordLangSupportedIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_is_thesaurus_lang_supported::codegen(
+ CompilerCB*,
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& argv,
+ expr& ann) const
+{
+ return new IsThesaurusLangSupportedIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_is_tokenizer_lang_supported::codegen(
+ CompilerCB*,
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& argv,
+ expr& ann) const
+{
+ return new IsTokenizerLangSupportedIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_stem::codegen(
+ CompilerCB*,
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& argv,
+ expr& ann) const
+{
+ return new StemIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_strip_diacritics::codegen(
+ CompilerCB*,
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& argv,
+ expr& ann) const
+{
+ return new StripDiacriticsIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_thesaurus_lookup::codegen(
+ CompilerCB*,
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& argv,
+ expr& ann) const
+{
+ return new ThesaurusLookupIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_tokenize_string::codegen(
+ CompilerCB*,
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& argv,
+ expr& ann) const
+{
+ return new TokenizeStringIterator(sctx, loc, argv);
+}
+
+#endif
+
+void populate_context_ft_module(static_context* sctx)
+{
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_current_lang,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","current-lang";),
+ GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE),
+ FunctionConsts::FULL_TEXT_CURRENT_LANG_0);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_host_lang,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","host-lang";),
+ GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE),
+ FunctionConsts::FULL_TEXT_HOST_LANG_0);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_is_stem_lang_supported,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","is-stem-lang-supported";),
+ GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
+ GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+ FunctionConsts::FULL_TEXT_IS_STEM_LANG_SUPPORTED_1);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_is_stop_word,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","is-stop-word";),
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+ FunctionConsts::FULL_TEXT_IS_STOP_WORD_1);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_is_stop_word,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","is-stop-word";),
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
+ GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+ FunctionConsts::FULL_TEXT_IS_STOP_WORD_2);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_is_stop_word_lang_supported,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","is-stop-word-lang-supported";),
+ GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
+ GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+ FunctionConsts::FULL_TEXT_IS_STOP_WORD_LANG_SUPPORTED_1);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_is_thesaurus_lang_supported,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","is-thesaurus-lang-supported";),
+ GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
+ GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+ FunctionConsts::FULL_TEXT_IS_THESAURUS_LANG_SUPPORTED_1);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_is_thesaurus_lang_supported,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","is-thesaurus-lang-supported";),
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
+ GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+ FunctionConsts::FULL_TEXT_IS_THESAURUS_LANG_SUPPORTED_2);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_is_tokenizer_lang_supported,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","is-tokenizer-lang-supported";),
+ GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
+ GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+ FunctionConsts::FULL_TEXT_IS_TOKENIZER_LANG_SUPPORTED_1);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_stem,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","stem";),
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_ONE),
+ FunctionConsts::FULL_TEXT_STEM_1);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_stem,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","stem";),
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_ONE),
+ FunctionConsts::FULL_TEXT_STEM_2);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_strip_diacritics,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","strip-diacritics";),
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_ONE),
+ FunctionConsts::FULL_TEXT_STRIP_DIACRITICS_1);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_thesaurus_lookup,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","thesaurus-lookup";),
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_PLUS),
+ FunctionConsts::FULL_TEXT_THESAURUS_LOOKUP_1);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_thesaurus_lookup,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","thesaurus-lookup";),
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_PLUS),
+ FunctionConsts::FULL_TEXT_THESAURUS_LOOKUP_2);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_thesaurus_lookup,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","thesaurus-lookup";),
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_PLUS),
+ FunctionConsts::FULL_TEXT_THESAURUS_LOOKUP_3);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_thesaurus_lookup,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","thesaurus-lookup";),
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_PLUS),
+ FunctionConsts::FULL_TEXT_THESAURUS_LOOKUP_4);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_thesaurus_lookup,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","thesaurus-lookup";),
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.INTEGER_TYPE_ONE,
+ GENV_TYPESYSTEM.INTEGER_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_PLUS),
+ FunctionConsts::FULL_TEXT_THESAURUS_LOOKUP_6);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_tokenize_string,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","tokenize-string";),
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_STAR),
+ FunctionConsts::FULL_TEXT_TOKENIZE_STRING_1);
+
+ }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+ {
+
+
+ DECL_WITH_KIND(sctx, full_text_tokenize_string,
+ (createQName("http://www.zorba-xquery.com/modules/full-text","","tokenize-string";),
+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
+ GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
+ GENV_TYPESYSTEM.STRING_TYPE_STAR),
+ FunctionConsts::FULL_TEXT_TOKENIZE_STRING_2);
+
+ }
+
+
+#endif
+}
+
+
+}
+
+
+
=== added file 'src/functions/pregenerated/func_ft_module.h'
--- src/functions/pregenerated/func_ft_module.h 1970-01-01 00:00:00 +0000
+++ src/functions/pregenerated/func_ft_module.h 2012-05-16 22:51:21 +0000
@@ -0,0 +1,225 @@
+/*
+ * Copyright 2006-2008 The FLWOR Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// ******************************************
+// * *
+// * THIS IS A GENERATED FILE. DO NOT EDIT! *
+// * SEE .xml FILE WITH SAME NAME *
+// * *
+// ******************************************
+
+
+#ifndef ZORBA_FUNCTIONS_FT_MODULE_H
+#define ZORBA_FUNCTIONS_FT_MODULE_H
+
+
+#include "common/shared_types.h"
+#include "functions/function_impl.h"
+
+
+namespace zorba {
+
+
+void populate_context_ft_module(static_context* sctx);
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:current-lang
+class full_text_current_lang : public function
+{
+public:
+ full_text_current_lang(const signature& sig, FunctionConsts::FunctionKind kind)
+ :
+ function(sig, kind)
+ {
+
+ }
+
+ CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:host-lang
+class full_text_host_lang : public function
+{
+public:
+ full_text_host_lang(const signature& sig, FunctionConsts::FunctionKind kind)
+ :
+ function(sig, kind)
+ {
+
+ }
+
+ CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:is-stem-lang-supported
+class full_text_is_stem_lang_supported : public function
+{
+public:
+ full_text_is_stem_lang_supported(const signature& sig, FunctionConsts::FunctionKind kind)
+ :
+ function(sig, kind)
+ {
+
+ }
+
+ CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:is-stop-word
+class full_text_is_stop_word : public function
+{
+public:
+ full_text_is_stop_word(const signature& sig, FunctionConsts::FunctionKind kind)
+ :
+ function(sig, kind)
+ {
+
+ }
+
+ CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:is-stop-word-lang-supported
+class full_text_is_stop_word_lang_supported : public function
+{
+public:
+ full_text_is_stop_word_lang_supported(const signature& sig, FunctionConsts::FunctionKind kind)
+ :
+ function(sig, kind)
+ {
+
+ }
+
+ CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:is-thesaurus-lang-supported
+class full_text_is_thesaurus_lang_supported : public function
+{
+public:
+ full_text_is_thesaurus_lang_supported(const signature& sig, FunctionConsts::FunctionKind kind)
+ :
+ function(sig, kind)
+ {
+
+ }
+
+ CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:is-tokenizer-lang-supported
+class full_text_is_tokenizer_lang_supported : public function
+{
+public:
+ full_text_is_tokenizer_lang_supported(const signature& sig, FunctionConsts::FunctionKind kind)
+ :
+ function(sig, kind)
+ {
+
+ }
+
+ CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:stem
+class full_text_stem : public function
+{
+public:
+ full_text_stem(const signature& sig, FunctionConsts::FunctionKind kind)
+ :
+ function(sig, kind)
+ {
+
+ }
+
+ CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:strip-diacritics
+class full_text_strip_diacritics : public function
+{
+public:
+ full_text_strip_diacritics(const signature& sig, FunctionConsts::FunctionKind kind)
+ :
+ function(sig, kind)
+ {
+
+ }
+
+ CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:thesaurus-lookup
+class full_text_thesaurus_lookup : public function
+{
+public:
+ full_text_thesaurus_lookup(const signature& sig, FunctionConsts::FunctionKind kind)
+ :
+ function(sig, kind)
+ {
+
+ }
+
+ CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:tokenize-string
+class full_text_tokenize_string : public function
+{
+public:
+ full_text_tokenize_string(const signature& sig, FunctionConsts::FunctionKind kind)
+ :
+ function(sig, kind)
+ {
+
+ }
+
+ CODEGEN_DECL();
+};
+#endif
+
+
+} //namespace zorba
+
+
+#endif
+/*
+ * Local variables:
+ * mode: c++
+ * End:
+ */
=== modified file 'src/functions/pregenerated/func_nodes.cpp'
--- src/functions/pregenerated/func_nodes.cpp 2012-04-24 12:39:38 +0000
+++ src/functions/pregenerated/func_nodes.cpp 2012-05-16 22:51:21 +0000
@@ -231,6 +231,16 @@
return new LeastCommonAncestor(sctx, loc, argv);
}
+PlanIter_t fn_path_3_0::codegen(
+ CompilerCB*,
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& argv,
+ expr& ann) const
+{
+ return new FnPathIterator(sctx, loc, argv);
+}
+
void populate_context_nodes(static_context* sctx)
{
{
@@ -356,6 +366,17 @@
DECL_WITH_KIND(sctx, fn_has_children_3_0,
(createQName("http://www.w3.org/2005/xpath-functions","","has-children";),
+ GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+ FunctionConsts::FN_HAS_CHILDREN_0);
+
+ }
+
+
+ {
+
+
+ DECL_WITH_KIND(sctx, fn_has_children_3_0,
+ (createQName("http://www.w3.org/2005/xpath-functions","","has-children";),
GENV_TYPESYSTEM.ANY_NODE_TYPE_QUESTION,
GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
FunctionConsts::FN_HAS_CHILDREN_1);
@@ -538,6 +559,29 @@
}
+
+ {
+
+
+ DECL_WITH_KIND(sctx, fn_path_3_0,
+ (createQName("http://www.w3.org/2005/xpath-functions","","path";),
+ GENV_TYPESYSTEM.STRING_TYPE_QUESTION),
+ FunctionConsts::FN_PATH_0);
+
+ }
+
+
+ {
+
+
+ DECL_WITH_KIND(sctx, fn_path_3_0,
+ (createQName("http://www.w3.org/2005/xpath-functions","","path";),
+ GENV_TYPESYSTEM.ANY_NODE_TYPE_QUESTION,
+ GENV_TYPESYSTEM.STRING_TYPE_QUESTION),
+ FunctionConsts::FN_PATH_1);
+
+ }
+
}
=== modified file 'src/functions/pregenerated/func_nodes.h'
--- src/functions/pregenerated/func_nodes.h 2012-04-24 12:39:38 +0000
+++ src/functions/pregenerated/func_nodes.h 2012-05-16 22:51:21 +0000
@@ -372,6 +372,21 @@
};
+//fn:path
+class fn_path_3_0 : public function
+{
+public:
+ fn_path_3_0(const signature& sig, FunctionConsts::FunctionKind kind)
+ :
+ function(sig, kind)
+ {
+theXQueryVersion = StaticContextConsts::xquery_version_3_0;
+ }
+
+ CODEGEN_DECL();
+};
+
+
} //namespace zorba
=== modified file 'src/functions/pregenerated/func_parse_fragment.cpp'
--- src/functions/pregenerated/func_parse_fragment.cpp 2012-04-24 12:39:38 +0000
+++ src/functions/pregenerated/func_parse_fragment.cpp 2012-05-16 22:51:21 +0000
@@ -38,7 +38,7 @@
std::vector<PlanIter_t>& argv,
expr& ann) const
{
- return new FnParseXmlFragmentIterator(sctx, loc, argv);
+ return new FnZorbaParseXmlFragmentIterator(sctx, loc, argv);
}
void populate_context_parse_fragment(static_context* sctx)
=== modified file 'src/functions/pregenerated/function_enum.h'
--- src/functions/pregenerated/function_enum.h 2012-04-24 12:39:38 +0000
+++ src/functions/pregenerated/function_enum.h 2012-05-16 22:51:21 +0000
@@ -22,6 +22,7 @@
// ******************************************
FN_NODE_NAME_0,
FN_NODE_NAME_1,
+ FN_NILLED_0,
FN_NILLED_1,
FN_STRING_0,
FN_STRING_1,
@@ -138,6 +139,25 @@
FN_ZORBA_FETCH_CONTENT_2,
FN_ZORBA_FETCH_CONTENT_TYPE_1,
FN_PUT_2,
+ FULL_TEXT_CURRENT_LANG_0,
+ FULL_TEXT_HOST_LANG_0,
+ FULL_TEXT_IS_STEM_LANG_SUPPORTED_1,
+ FULL_TEXT_IS_STOP_WORD_1,
+ FULL_TEXT_IS_STOP_WORD_2,
+ FULL_TEXT_IS_STOP_WORD_LANG_SUPPORTED_1,
+ FULL_TEXT_IS_THESAURUS_LANG_SUPPORTED_1,
+ FULL_TEXT_IS_THESAURUS_LANG_SUPPORTED_2,
+ FULL_TEXT_IS_TOKENIZER_LANG_SUPPORTED_1,
+ FULL_TEXT_STEM_1,
+ FULL_TEXT_STEM_2,
+ FULL_TEXT_STRIP_DIACRITICS_1,
+ FULL_TEXT_THESAURUS_LOOKUP_1,
+ FULL_TEXT_THESAURUS_LOOKUP_2,
+ FULL_TEXT_THESAURUS_LOOKUP_3,
+ FULL_TEXT_THESAURUS_LOOKUP_4,
+ FULL_TEXT_THESAURUS_LOOKUP_6,
+ FULL_TEXT_TOKENIZE_STRING_1,
+ FULL_TEXT_TOKENIZE_STRING_2,
FN_FUNCTION_NAME_1,
FN_FUNCTION_ARITY_1,
FN_PARTIAL_APPLY_2,
@@ -232,6 +252,7 @@
FN_LANG_2,
FN_NUMBER_0,
FN_NUMBER_1,
+ FN_HAS_CHILDREN_0,
FN_HAS_CHILDREN_1,
FN_INNERMOST_1,
FN_OUTERMOST_1,
@@ -247,6 +268,8 @@
FN_ZORBA_NODE_PRECEDING_SIBLING_OF_2,
FN_ZORBA_NODE_LEVEL_1,
FN_ZORBA_NODE_LEAST_COMMON_ANCESTOR_2,
+ FN_PATH_0,
+ FN_PATH_1,
FN_ABS_1,
FN_CEILING_1,
FN_FLOOR_1,
=== modified file 'src/runtime/accessors/accessors_impl.cpp'
--- src/runtime/accessors/accessors_impl.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/accessors/accessors_impl.cpp 2012-05-16 22:51:21 +0000
@@ -78,15 +78,23 @@
if (consumeNext(inNode, theChildren[0].getp(), planState))
{
- if (inNode->isNode()) {
+ if (inNode->isNode())
+ {
result = inNode->getNilled();
STACK_PUSH(result != NULL, state);
- } else
+ }
+ else
+ {
throw XQUERY_EXCEPTION(
err::XPTY0004,
ERROR_PARAMS( ZED( FnNilledArgNotNode ) ),
ERROR_LOC( loc )
);
+ }
+ }
+ else
+ {
+ STACK_PUSH(false, state);
}
STACK_END (state);
=== modified file 'src/runtime/collections/collections_impl.cpp'
--- src/runtime/collections/collections_impl.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/collections/collections_impl.cpp 2012-05-16 22:51:21 +0000
@@ -2504,122 +2504,122 @@
/*******************************************************************************
14.8.5 fn:uri-collection
********************************************************************************/
- FnURICollectionIteratorState::~FnURICollectionIteratorState()
- {
- if(theIterator != NULL)
- {
- if(theIteratorOpened)
- {
- theIterator->close();
- theIteratorOpened = false;
- }
- theIterator = NULL;
- }
- }
-
- void FnURICollectionIteratorState::init(PlanState& planState)
- {
- PlanIteratorState::init(planState);
- theIterator = NULL;
- }
-
- void FnURICollectionIteratorState::reset(PlanState& planState)
- {
- PlanIteratorState::reset(planState);
-
- if(theIterator != NULL)
- {
- if(theIteratorOpened)
- {
- theIterator->close();
- theIteratorOpened = false;
- }
- theIterator = NULL;
- }
- }
-
- bool FnURICollectionIterator::nextImpl(store::Item_t& result, PlanState& planState) const
- {
- store::Item_t lURI, resolvedURIItem, lIte;
- store::Collection_t coll;
- std::auto_ptr<internal::Resource> lResource;
- internal::CollectionResource* lCollResource;
- zstring resolvedURIString;
- zstring lErrorMessage;
- zstring docuri;
-
- FnURICollectionIteratorState* state;
- DEFAULT_STACK_INIT(FnURICollectionIteratorState, state, planState);
-
- if(theChildren.size() == 1 &&
- consumeNext(lURI, theChildren[0].getp(),planState))
- {
- try
- {
- resolvedURIString= theSctx->resolve_relative_uri(lURI->getStringValue());
- }
- catch (ZorbaException const&)
- {
- throw XQUERY_EXCEPTION(
- err::FODC0004,
- ERROR_PARAMS(lURI->getStringValue(), ZED( BadAnyURI ) ),
- ERROR_LOC( loc )
- );
- }
- }
- else
- {
- resolvedURIItem = planState.theGlobalDynCtx->get_default_collection();
-
- if ( NULL == resolvedURIItem )
- throw XQUERY_EXCEPTION(
- err::FODC0002,
- ERROR_PARAMS( ZED( DefaultCollection), ZED( NotDefInDynamicCtx ) ),
- ERROR_LOC( loc )
- );
-
- resolvedURIString = theSctx->resolve_relative_uri(resolvedURIItem->getStringValue());
- }
-
- lResource = theSctx->resolve_uri(resolvedURIString,
- internal::EntityData::COLLECTION,
- lErrorMessage);
-
- lCollResource = dynamic_cast<internal::CollectionResource*>(lResource.get());
-
- if( lCollResource == 0 || !(coll = lCollResource->getCollection()) )
- {
- throw XQUERY_EXCEPTION(
- err::FODC0002,
- ERROR_PARAMS( resolvedURIString, lErrorMessage ),
- ERROR_LOC( loc )
- );
- }
-
- // return collection nodes
- state->theIterator = coll->getIterator();
- ZORBA_ASSERT(state->theIterator != NULL);
- state->theIterator->open();
- state->theIteratorOpened = true;
-
- //return the DocumentURI of the Collection
- while(state->theIterator->next(lIte))
- {
- lIte->getDocumentURI(docuri);
- if(!docuri.empty())
- {
- STACK_PUSH(GENV_ITEMFACTORY->createAnyURI(result, docuri), state);
- }
- }
-
- //close iterator
- state->theIterator->close();
- state->theIteratorOpened = false;
-
- STACK_PUSH(false, state);
- STACK_END(state);
-
- }
+FnURICollectionIteratorState::~FnURICollectionIteratorState()
+{
+ if(theIterator != NULL)
+ {
+ if(theIteratorOpened)
+ {
+ theIterator->close();
+ theIteratorOpened = false;
+ }
+ theIterator = NULL;
+ }
+}
+
+void FnURICollectionIteratorState::init(PlanState& planState)
+{
+ PlanIteratorState::init(planState);
+ theIterator = NULL;
+}
+
+void FnURICollectionIteratorState::reset(PlanState& planState)
+{
+ PlanIteratorState::reset(planState);
+
+ if(theIterator != NULL)
+ {
+ if(theIteratorOpened)
+ {
+ theIterator->close();
+ theIteratorOpened = false;
+ }
+ theIterator = NULL;
+ }
+}
+
+bool FnURICollectionIterator::nextImpl(store::Item_t& result, PlanState& planState) const
+{
+ store::Item_t lURI, resolvedURIItem, lIte;
+ store::Collection_t coll;
+ std::auto_ptr<internal::Resource> lResource;
+ internal::CollectionResource* lCollResource;
+ zstring resolvedURIString;
+ zstring lErrorMessage;
+ zstring docuri;
+
+ FnURICollectionIteratorState* state;
+ DEFAULT_STACK_INIT(FnURICollectionIteratorState, state, planState);
+
+ if(theChildren.size() == 1 &&
+ consumeNext(lURI, theChildren[0].getp(),planState))
+ {
+ try
+ {
+ resolvedURIString= theSctx->resolve_relative_uri(lURI->getStringValue());
+ }
+ catch (ZorbaException const&)
+ {
+ throw XQUERY_EXCEPTION(
+ err::FODC0004,
+ ERROR_PARAMS(lURI->getStringValue(), ZED( BadAnyURI ) ),
+ ERROR_LOC( loc )
+ );
+ }
+ }
+ else
+ {
+ resolvedURIItem = planState.theGlobalDynCtx->get_default_collection();
+
+ if ( NULL == resolvedURIItem )
+ throw XQUERY_EXCEPTION(
+ err::FODC0002,
+ ERROR_PARAMS( ZED( DefaultCollection), ZED( NotDefInDynamicCtx ) ),
+ ERROR_LOC( loc )
+ );
+
+ resolvedURIString = theSctx->resolve_relative_uri(resolvedURIItem->getStringValue());
+ }
+
+ lResource = theSctx->resolve_uri(resolvedURIString,
+ internal::EntityData::COLLECTION,
+ lErrorMessage);
+
+ lCollResource = dynamic_cast<internal::CollectionResource*>(lResource.get());
+
+ if( lCollResource == 0 || !(coll = lCollResource->getCollection()) )
+ {
+ throw XQUERY_EXCEPTION(
+ err::FODC0002,
+ ERROR_PARAMS( resolvedURIString, lErrorMessage ),
+ ERROR_LOC( loc )
+ );
+ }
+
+ // return collection nodes
+ state->theIterator = coll->getIterator();
+ ZORBA_ASSERT(state->theIterator != NULL);
+ state->theIterator->open();
+ state->theIteratorOpened = true;
+
+ //return the DocumentURI of the Collection
+ while(state->theIterator->next(lIte))
+ {
+ lIte->getDocumentURI(docuri);
+ if(!docuri.empty())
+ {
+ STACK_PUSH(GENV_ITEMFACTORY->createAnyURI(result, docuri), state);
+ }
+ }
+
+ //close iterator
+ state->theIterator->close();
+ state->theIteratorOpened = false;
+
+ STACK_PUSH(false, state);
+ STACK_END(state);
+
+}
} // namespace zorba
=== modified file 'src/runtime/collections/pregenerated/collections.h'
--- src/runtime/collections/pregenerated/collections.h 2012-04-24 12:39:38 +0000
+++ src/runtime/collections/pregenerated/collections.h 2012-05-16 22:51:21 +0000
@@ -1469,9 +1469,9 @@
/**
*
- * Returns a sequence of xs:anyURI values representing the document URIs of the
- * documents in a collection.
- *
+ * Returns a sequence of xs:anyURI values representing the document URIs of the
+ * documents in a collection.
+ *
* Author: Zorba Team
*/
class FnURICollectionIteratorState : public PlanIteratorState
=== modified file 'src/runtime/core/apply_updates.cpp'
--- src/runtime/core/apply_updates.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/core/apply_updates.cpp 2012-05-16 22:51:21 +0000
@@ -92,19 +92,24 @@
store::Item_t item;
ulong numItems = 0;
- std::auto_ptr<store::PUL> pul;
+ store::PUL_t pul;
ApplyIteratorState* state;
DEFAULT_STACK_INIT(ApplyIteratorState, state, planState);
- pul.reset(GENV_ITEMFACTORY->createPendingUpdateList());
-
// Note: updating expr might not return a pul because of vacuous exprs
while (consumeNext(item, theChild, planState))
{
if (item->isPul())
{
- pul->mergeUpdates(item);
+ if (pul)
+ {
+ pul->mergeUpdates(item);
+ }
+ else
+ {
+ pul.transfer(item);
+ }
}
else if (!theDiscardXDM)
{
@@ -113,7 +118,8 @@
}
}
- apply_updates(ccb, gdctx, theSctx, pul.get(), loc);
+ if(pul)
+ apply_updates(ccb, gdctx, theSctx, pul, loc);
state->theXDMIte = state->theXDMItems.begin();
state->theXDMEnd = state->theXDMItems.end();
=== modified file 'src/runtime/full_text/CMakeLists.txt'
--- src/runtime/full_text/CMakeLists.txt 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/CMakeLists.txt 2012-05-16 22:51:21 +0000
@@ -13,6 +13,7 @@
# limitations under the License.
SET(FULLTEXT_SRCS
+ ft_util.cpp
ft_match.cpp
ft_query_item.cpp
ft_single_token_iterator.cpp
@@ -40,6 +41,7 @@
thesaurus.cpp
tokenizer.cpp
default_tokenizer.cpp
+ ft_module.cpp
)
IF (ZORBA_NO_ICU)
@@ -51,5 +53,5 @@
ADD_SRC_SUBFOLDER(FULLTEXT_SRCS stemmer LIBSTEMMER_SRCS)
IF (ZORBA_WITH_FILE_ACCESS)
- ADD_SRC_SUBFOLDER(FULLTEXT_SRCS thesauri THESAURUS_SRCS)
+ ADD_SRC_SUBFOLDER(FULLTEXT_SRCS thesauri THESAURUS_SRCS)
ENDIF (ZORBA_WITH_FILE_ACCESS)
=== modified file 'src/runtime/full_text/apply.cpp'
--- src/runtime/full_text/apply.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/apply.cpp 2012-05-16 22:51:21 +0000
@@ -26,13 +26,14 @@
#include "diagnostics/dict.h"
#include "diagnostics/xquery_diagnostics.h"
#include "store/api/item.h"
+#include "store/api/item_factory.h"
#include "store/api/store.h"
-#include "store/api/item_factory.h"
#include "system/globalenv.h"
#include "util/cxx_util.h"
#include "util/indent.h"
#include "util/stl_util.h"
#include "zorbamisc/ns_consts.h"
+#include "zorbautils/locale.h"
#ifndef NDEBUG
# include "system/properties.h"
@@ -1184,11 +1185,10 @@
{
}
- void operator()( char const *utf8_s, size_type utf8_len, size_type,
- size_type, size_type, void* ) {
- FTToken const t( utf8_s, (int)utf8_len, token_no_, lang_ );
- tokens_.push_back( t );
- }
+ // inherited
+ void item( Item const&, bool );
+ void token( char const*, size_type, iso639_1::type, size_type, size_type,
+ size_type, Item const* );
private:
FTTokenSeqIterator::FTTokens &tokens_;
@@ -1196,51 +1196,76 @@
iso639_1::type const lang_;
};
+void thesaurus_callback::item( Item const&, bool ) {
+ // out-of-line since it's virtual
+}
+
+void thesaurus_callback::token( char const *utf8_s, size_type utf8_len,
+ iso639_1::type, size_type, size_type,
+ size_type, Item const* ) {
+ FTToken const t( utf8_s, (int)utf8_len, token_no_, lang_ );
+ tokens_.push_back( t );
+}
+
} // anonymous namespace
void ftcontains_visitor::
-lookup_thesaurus( ftthesaurus_id const &tid, zstring const &query_phrase,
+lookup_thesaurus( ftthesaurus_id const &t_id, zstring const &query_phrase,
FTToken const &qt0, query_item_star_t &result ) {
ft_int at_least, at_most;
- if ( ftrange const *const levels = tid.get_levels() )
+ if ( ftrange const *const levels = t_id.get_levels() )
eval_ftrange( *levels, &at_least, &at_most );
else
at_least = 0, at_most = numeric_limits<ft_int>::max();
- zstring const &uri = tid.get_uri();
+ zstring const &uri = t_id.get_uri();
zstring error_msg;
auto_ptr<internal::Resource> rsrc = static_ctx_.resolve_uri(
- uri, internal::ThesaurusEntityData( qt0.lang() ), error_msg
+ uri, internal::EntityData::THESAURUS, error_msg
);
if ( !rsrc.get() )
throw XQUERY_EXCEPTION( err::FTST0018, ERROR_PARAMS( uri ) );
- internal::Thesaurus::ptr thesaurus(
- dynamic_cast<internal::Thesaurus*>( rsrc.release() )
- );
- if ( !thesaurus )
- throw XQUERY_EXCEPTION( err::FTST0018, ERROR_PARAMS( uri ) );
-
- internal::Thesaurus::iterator::ptr tresult(
+ internal::ThesaurusProvider const *const t_provider =
+ dynamic_cast<internal::ThesaurusProvider const*>( rsrc.get() );
+ ZORBA_ASSERT( t_provider );
+
+ internal::Thesaurus::ptr thesaurus;
+ if ( !t_provider->getThesaurus( qt0.lang(), &thesaurus ) )
+ throw XQUERY_EXCEPTION(
+ err::FTST0009,
+ ERROR_PARAMS(
+ iso639_1::string_of[ qt0.lang() ], ZED( FTST0009_BadThesaurusLang )
+ )
+ );
+
+ internal::Thesaurus::iterator::ptr t_synonyms(
thesaurus->lookup(
- query_phrase, tid.get_relationship(), at_least, at_most
+ query_phrase, t_id.get_relationship(), at_least, at_most
)
);
- if ( !tresult )
+ if ( !t_synonyms )
return;
FTTokenSeqIterator::FTTokens synonyms;
thesaurus_callback cb( qt0.pos(), qt0.lang(), synonyms );
- Tokenizer::Numbers tno;
- Tokenizer::ptr tokenizer(
- GENV_STORE.getTokenizerProvider()->getTokenizer( qt0.lang(), tno )
- );
+ Tokenizer::Numbers t_num;
+ TokenizerProvider const *const provider = GENV_STORE.getTokenizerProvider();
+ ZORBA_ASSERT( provider );
+ Tokenizer::ptr tokenizer;
+ if ( !provider->getTokenizer( qt0.lang(), &t_num, &tokenizer ) )
+ throw XQUERY_EXCEPTION(
+ err::FTST0009,
+ ERROR_PARAMS(
+ iso639_1::string_of[ qt0.lang() ], ZED( FTST0009_BadTokenizerLang )
+ )
+ );
- for ( zstring synonym; tresult->next( &synonym ); ) {
+ for ( zstring synonym; t_synonyms->next( &synonym ); ) {
synonyms.clear();
- tokenizer->tokenize(
+ tokenizer->tokenize_string(
synonym.data(), synonym.size(), qt0.lang(), false, cb
);
query_item_t const query_item( new FTTokenSeqIterator( synonyms ) );
=== added file 'src/runtime/full_text/ft_module_impl.cpp'
--- src/runtime/full_text/ft_module_impl.cpp 1970-01-01 00:00:00 +0000
+++ src/runtime/full_text/ft_module_impl.cpp 2012-05-16 22:51:21 +0000
@@ -0,0 +1,819 @@
+/*
+ * Copyright 2006-2008 The FLWOR Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <zorba/config.h>
+
+// This entire file should be effectively skipped when building with
+// ZORBA_NO_FULL_TEXT. We can't prevent it from being compiled, but we
+// can prevent it from having any real content.
+#ifndef ZORBA_NO_FULL_TEXT
+
+# include <limits>
+# include <typeinfo>
+
+# include <zorba/diagnostic_list.h>
+
+# include "api/unmarshaller.h"
+# include "context/namespace_context.h"
+# include "context/static_context.h"
+# include "diagnostics/assert.h"
+# include "diagnostics/xquery_diagnostics.h"
+# include "store/api/index.h"
+# include "store/api/item.h"
+# include "store/api/item_factory.h"
+# include "store/api/iterator.h"
+# include "store/api/store.h"
+# include "system/globalenv.h"
+# include "types/casting.h"
+# include "types/typeimpl.h"
+# include "types/typeops.h"
+# include "util/utf8_util.h"
+# include "zorbatypes/URI.h"
+# include "zorbautils/locale.h"
+
+# include "ft_stop_words_set.h"
+# include "ft_token_seq_iterator.h"
+# include "ft_util.h"
+# include "thesaurus.h"
+
+#include "runtime/full_text/ft_module.h"
+
+using namespace std;
+using namespace zorba::locale;
+
+namespace zorba {
+
+///////////////////////////////////////////////////////////////////////////////
+
+inline iso639_1::type get_lang_from( static_context const *sctx ) {
+ iso639_1::type const lang = get_lang_from( sctx->get_match_options() );
+ return lang ? lang : get_host_lang();
+}
+
+static iso639_1::type get_lang_from( store::Item_t lang_item,
+ QueryLoc const &loc ) {
+ zstring lang_string;
+ lang_item->getStringValue2( lang_string );
+
+ if ( !GenericCast::instance()->castableToLanguage( lang_string ) )
+ throw XQUERY_EXCEPTION(
+ err::XPTY0004,
+ ERROR_PARAMS(
+ ZED( BadType_23o ), lang_string, ZED( NoCastTo_45o ), "xs:language"
+ ),
+ ERROR_LOC( loc )
+ );
+ if ( iso639_1::type const lang = find_lang( lang_string.c_str() ) )
+ return lang;
+ throw XQUERY_EXCEPTION(
+ err::FTST0009, ERROR_PARAMS( lang_string ), ERROR_LOC( loc )
+ );
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+bool CurrentLangIterator::nextImpl( store::Item_t &result,
+ PlanState &plan_state ) const {
+ static_context const *const sctx = getStaticContext();
+ ZORBA_ASSERT( sctx );
+ iso639_1::type const lang = get_lang_from( sctx );
+ zstring lang_string( iso639_1::string_of[ lang ] );
+
+ PlanIteratorState *state;
+ DEFAULT_STACK_INIT( PlanIteratorState, state, plan_state );
+
+ GENV_ITEMFACTORY->createLanguage( result, lang_string );
+ STACK_PUSH( true, state );
+
+ STACK_END( state );
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+bool HostLangIterator::nextImpl( store::Item_t &result,
+ PlanState &plan_state ) const {
+ iso639_1::type const lang = get_host_lang();
+ zstring lang_string = iso639_1::string_of[ lang ];
+
+ PlanIteratorState *state;
+ DEFAULT_STACK_INIT( PlanIteratorState, state, plan_state );
+
+ GENV_ITEMFACTORY->createLanguage( result, lang_string );
+ STACK_PUSH( true, state );
+
+ STACK_END( state );
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+bool IsStemLangSupportedIterator::nextImpl( store::Item_t &result,
+ PlanState &plan_state ) const {
+ bool is_supported;
+ store::Item_t item;
+
+ PlanIteratorState *state;
+ DEFAULT_STACK_INIT( PlanIteratorState, state, plan_state );
+
+ consumeNext( item, theChildren[0], plan_state );
+ try {
+ internal::StemmerProvider const *const provider =
+ GENV_STORE.getStemmerProvider();
+ is_supported = provider->getStemmer( get_lang_from( item, loc ) );
+ }
+ catch ( XQueryException const &e ) {
+ if ( e.diagnostic() != err::FTST0009 )
+ throw;
+ is_supported = false;
+ }
+
+ GENV_ITEMFACTORY->createBoolean( result, is_supported );
+ STACK_PUSH( true, state );
+
+ STACK_END( state );
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+bool IsStopWordIterator::nextImpl( store::Item_t &result,
+ PlanState &plan_state ) const {
+ store::Item_t item;
+ iso639_1::type lang;
+ static_context const *const sctx = getStaticContext();
+ ZORBA_ASSERT( sctx );
+ ft_stop_words_set::ptr stop_words;
+ zstring word;
+
+ PlanIteratorState *state;
+ DEFAULT_STACK_INIT( PlanIteratorState, state, plan_state );
+
+ lang = get_lang_from( sctx );
+
+ consumeNext( item, theChildren[0], plan_state );
+ item->getStringValue2( word );
+
+ if ( theChildren.size() > 1 ) {
+ consumeNext( item, theChildren[1], plan_state );
+ lang = get_lang_from( item, loc );
+ }
+
+ stop_words.reset( ft_stop_words_set::get_default( lang ) );
+ if ( !stop_words )
+ throw XQUERY_EXCEPTION(
+ err::FTST0009,
+ ERROR_PARAMS(
+ iso639_1::string_of[ lang ], ZED( FTST0009_BadStopWordsLang )
+ ),
+ ERROR_LOC( loc )
+ );
+ GENV_ITEMFACTORY->createBoolean( result, stop_words->contains( word ) );
+ STACK_PUSH( true, state );
+
+ STACK_END( state );
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+bool IsStopWordLangSupportedIterator::nextImpl( store::Item_t &result,
+ PlanState &plan_state ) const {
+ bool is_supported;
+ store::Item_t item;
+
+ PlanIteratorState *state;
+ DEFAULT_STACK_INIT( PlanIteratorState, state, plan_state );
+
+ consumeNext( item, theChildren[0], plan_state );
+ try {
+ is_supported = ft_stop_words_set::get_default( get_lang_from( item, loc ) );
+ }
+ catch ( XQueryException const &e ) {
+ if ( e.diagnostic() != err::FTST0009 )
+ throw;
+ is_supported = false;
+ }
+
+ GENV_ITEMFACTORY->createBoolean( result, is_supported );
+ STACK_PUSH( true, state );
+
+ STACK_END( state );
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+bool IsThesaurusLangSupportedIterator::nextImpl( store::Item_t &result,
+ PlanState &plan_state ) const {
+ bool is_supported;
+ store::Item_t item;
+ zstring uri;
+
+ PlanIteratorState *state;
+ DEFAULT_STACK_INIT( PlanIteratorState, state, plan_state );
+
+ consumeNext( item, theChildren[0], plan_state );
+ if ( theChildren.size() > 1 ) {
+ item->getStringValue2( uri );
+ consumeNext( item, theChildren[1], plan_state );
+ } else {
+ uri = "##default";
+ }
+
+ try {
+ iso639_1::type const lang = get_lang_from( item, loc );
+ static_context const *const sctx = getStaticContext();
+ ZORBA_ASSERT( sctx );
+
+ zstring error_msg;
+ auto_ptr<internal::Resource> rsrc = sctx->resolve_uri(
+ uri, internal::EntityData::THESAURUS, error_msg
+ );
+ if ( !rsrc.get() )
+ throw XQUERY_EXCEPTION(
+ err::FTST0018, ERROR_PARAMS( uri ), ERROR_LOC( loc )
+ );
+#if 0
+ if ( !error_msg.empty() )
+ cerr << "error_msg=" << error_msg << endl;
+#endif
+ internal::ThesaurusProvider const *const provider =
+ dynamic_cast<internal::ThesaurusProvider const*>( rsrc.get() );
+ ZORBA_ASSERT( provider );
+ is_supported = provider->getThesaurus( lang );
+ }
+ catch ( XQueryException const &e ) {
+ if ( e.diagnostic() != err::FTST0009 /* lang not supported by Zorba */ )
+ throw;
+ is_supported = false;
+ }
+
+ GENV_ITEMFACTORY->createBoolean( result, is_supported );
+ STACK_PUSH( true, state );
+
+ STACK_END( state );
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+bool IsTokenizerLangSupportedIterator::nextImpl( store::Item_t &result,
+ PlanState &plan_state ) const {
+ bool is_supported;
+ store::Item_t item;
+
+ PlanIteratorState *state;
+ DEFAULT_STACK_INIT( PlanIteratorState, state, plan_state );
+
+ consumeNext( item, theChildren[0], plan_state );
+ try {
+ TokenizerProvider const *const p = GENV_STORE.getTokenizerProvider();
+ is_supported = p && p->getTokenizer( get_lang_from( item, loc ) );
+ }
+ catch ( XQueryException const &e ) {
+ if ( e.diagnostic() != err::FTST0009 )
+ throw;
+ is_supported = false;
+ }
+
+ GENV_ITEMFACTORY->createBoolean( result, is_supported );
+ STACK_PUSH( true, state );
+
+ STACK_END( state );
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+bool StemIterator::nextImpl( store::Item_t &result,
+ PlanState &plan_state ) const {
+ store::Item_t item;
+ iso639_1::type lang;
+ internal::StemmerProvider const *provider;
+ static_context const *sctx;
+ internal::Stemmer::ptr stemmer;
+ zstring word, stem;
+
+ PlanIteratorState *state;
+ DEFAULT_STACK_INIT( PlanIteratorState, state, plan_state );
+
+ sctx = getStaticContext();
+ ZORBA_ASSERT( sctx );
+ lang = get_lang_from( sctx );
+
+ consumeNext( item, theChildren[0], plan_state );
+ item->getStringValue2( word );
+ utf8::to_lower( word );
+
+ if ( theChildren.size() > 1 ) {
+ consumeNext( item, theChildren[1], plan_state );
+ lang = get_lang_from( item, loc );
+ }
+
+ provider = GENV_STORE.getStemmerProvider();
+ ZORBA_ASSERT( provider );
+ if ( provider->getStemmer( lang, &stemmer ) ) {
+ stemmer->stem( word, lang, &stem );
+ GENV_ITEMFACTORY->createString( result, stem );
+ STACK_PUSH( true, state );
+ } else {
+ throw XQUERY_EXCEPTION(
+ err::FTST0009,
+ ERROR_PARAMS(
+ iso639_1::string_of[ lang ], ZED( FTST0009_BadStemmerLang )
+ ),
+ ERROR_LOC( loc )
+ );
+ }
+
+ STACK_END( state );
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+bool StripDiacriticsIterator::nextImpl( store::Item_t &result,
+ PlanState &plan_state ) const {
+ store::Item_t item;
+ zstring phrase, stripped_phrase;
+
+ PlanIteratorState *state;
+ DEFAULT_STACK_INIT( PlanIteratorState, state, plan_state );
+
+ consumeNext( item, theChildren[0], plan_state );
+ item->getStringValue2( phrase );
+ utf8::strip_diacritics( phrase, &stripped_phrase );
+ GENV_ITEMFACTORY->createString( result, stripped_phrase );
+ STACK_PUSH( true, state );
+
+ STACK_END( state );
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+bool ThesaurusLookupIterator::nextImpl( store::Item_t &result,
+ PlanState &plan_state ) const {
+ zstring error_msg;
+ store::Item_t item;
+ iso639_1::type lang;
+ auto_ptr<internal::Resource> rsrc;
+ zstring uri = "##default";
+ static_context const *sctx;
+ zstring synonym;
+ internal::ThesaurusProvider const *provider;
+
+ ThesaurusLookupIteratorState *state;
+ DEFAULT_STACK_INIT( ThesaurusLookupIteratorState, state, plan_state );
+
+ sctx = getStaticContext();
+ ZORBA_ASSERT( sctx );
+ lang = get_lang_from( sctx );
+ state->at_least_ = 0;
+ state->at_most_ = numeric_limits<internal::Thesaurus::level_type>::max();
+
+ if ( theChildren.size() == 1 ) {
+ consumeNext( item, theChildren[0], plan_state );
+ item->getStringValue2( state->phrase_ );
+ } else if ( theChildren.size() > 1 ) {
+ consumeNext( item, theChildren[0], plan_state );
+ item->getStringValue2( uri );
+ consumeNext( item, theChildren[1], plan_state );
+ item->getStringValue2( state->phrase_ );
+ if ( theChildren.size() > 2 ) {
+ consumeNext( item, theChildren[2], plan_state );
+ lang = get_lang_from( item, loc );
+ if ( theChildren.size() > 3 ) {
+ consumeNext( item, theChildren[3], plan_state );
+ item->getStringValue2( state->relationship_ );
+ if ( theChildren.size() > 4 ) {
+ ZORBA_ASSERT( theChildren.size() == 6 );
+ consumeNext( item, theChildren[4], plan_state );
+ state->at_least_ = to_ft_int( item->getIntegerValue() );
+ consumeNext( item, theChildren[5], plan_state );
+ state->at_most_ = to_ft_int( item->getIntegerValue() );
+ }
+ }
+ }
+ }
+
+ rsrc = sctx->resolve_uri(
+ uri, internal::EntityData::THESAURUS, error_msg
+ );
+ if ( !rsrc.get() )
+ throw XQUERY_EXCEPTION(
+ err::FTST0018, ERROR_PARAMS( uri ), ERROR_LOC( loc )
+ );
+
+ provider = dynamic_cast<internal::ThesaurusProvider const*>( rsrc.get() );
+ ZORBA_ASSERT( provider );
+ if ( !provider->getThesaurus( lang, &state->thesaurus_ ) )
+ throw XQUERY_EXCEPTION(
+ err::FTST0009,
+ ERROR_PARAMS(
+ iso639_1::string_of[ lang ], ZED( FTST0009_BadThesaurusLang )
+ ),
+ ERROR_LOC( loc )
+ );
+
+ state->tresult_ = std::move(
+ state->thesaurus_->lookup(
+ state->phrase_, state->relationship_, state->at_least_, state->at_most_
+ )
+ );
+ ZORBA_ASSERT( state->tresult_.get() );
+
+ while ( state->tresult_->next( &synonym ) ) {
+ GENV_ITEMFACTORY->createString( result, synonym );
+ STACK_PUSH( true, state );
+ }
+
+ STACK_END( state );
+}
+
+void ThesaurusLookupIterator::resetImpl( PlanState &plan_state ) const {
+ NaryBaseIterator<ThesaurusLookupIterator,ThesaurusLookupIteratorState>::
+ resetImpl( plan_state );
+ ThesaurusLookupIteratorState *const state =
+ StateTraitsImpl<ThesaurusLookupIteratorState>::getState(
+ plan_state, this->theStateOffset
+ );
+ state->tresult_ = std::move(
+ state->thesaurus_->lookup(
+ state->phrase_, state->relationship_, state->at_least_, state->at_most_
+ )
+ );
+ ZORBA_ASSERT( state->tresult_.get() );
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+TokenizeIterator::TokenizeIterator(
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& children)
+ : NaryBaseIterator<TokenizeIterator, TokenizeIteratorState>(sctx, loc, children)
+{
+ initMembers();
+}
+
+void TokenizeIterator::serialize( ::zorba::serialization::Archiver& ar)
+{
+ serialize_baseclass(ar,
+ (NaryBaseIterator<TokenizeIterator, TokenizeIteratorState>*)this);
+ if (!ar.is_serializing_out())
+ {
+ initMembers();
+ }
+}
+
+void TokenizeIterator::initMembers() {
+ GENV_ITEMFACTORY->createQName(
+ token_qname_, static_context::ZORBA_FULL_TEXT_FN_NS, "", "token");
+
+ GENV_ITEMFACTORY->createQName(
+ lang_qname_, "", "", "lang");
+
+ GENV_ITEMFACTORY->createQName(
+ para_qname_, "", "", "paragraph");
+
+ GENV_ITEMFACTORY->createQName(
+ sent_qname_, "", "", "sentence");
+
+ GENV_ITEMFACTORY->createQName(
+ value_qname_, "", "", "value");
+
+ GENV_ITEMFACTORY->createQName(
+ ref_qname_, "", "", "node-ref");
+}
+
+bool TokenizeIterator::nextImpl( store::Item_t &result,
+ PlanState &plan_state ) const {
+ store::Item_t node_name, attr_node;
+ zstring base_uri;
+ store::Item_t item;
+ iso639_1::type lang;
+ Tokenizer::Numbers no;
+ store::NsBindings const ns_bindings;
+ static_context const *const sctx = getStaticContext();
+ ZORBA_ASSERT( sctx );
+ TokenizerProvider const *tokenizer_provider;
+ store::Item_t type_name;
+ zstring value_string;
+
+ TokenizeIteratorState *state;
+ DEFAULT_STACK_INIT( TokenizeIteratorState, state, plan_state );
+
+ lang = get_lang_from( sctx );
+
+ if ( consumeNext( state->doc_item_, theChildren[0], plan_state ) ) {
+ if ( theChildren.size() > 1 ) {
+ consumeNext( item, theChildren[1], plan_state );
+ lang = get_lang_from( item, loc );
+ }
+
+ tokenizer_provider = GENV_STORE.getTokenizerProvider();
+ ZORBA_ASSERT( tokenizer_provider );
+ state->doc_tokens_ =
+ state->doc_item_->getTokens( *tokenizer_provider, no, lang );
+
+ while ( state->doc_tokens_->hasNext() ) {
+ FTToken const *token;
+ token = state->doc_tokens_->next();
+ ZORBA_ASSERT( token );
+
+ base_uri = static_context::ZORBA_FULL_TEXT_FN_NS;
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_QNAME;
+ node_name = token_qname_;
+ GENV_ITEMFACTORY->createElementNode(
+ result, nullptr, node_name, type_name, false, false,
+ ns_bindings, base_uri
+ );
+
+ if ( token->lang() ) {
+ value_string = iso639_1::string_of[ token->lang() ];
+ GENV_ITEMFACTORY->createString( item, value_string );
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_QNAME;
+ node_name = lang_qname_;
+ GENV_ITEMFACTORY->createAttributeNode(
+ attr_node, result, node_name, type_name, item
+ );
+ }
+
+ ztd::to_string( token->para(), &value_string );
+ GENV_ITEMFACTORY->createString( item, value_string );
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_QNAME;
+ node_name = para_qname_;
+ GENV_ITEMFACTORY->createAttributeNode(
+ attr_node, result, node_name, type_name, item
+ );
+
+ ztd::to_string( token->sent(), &value_string );
+ GENV_ITEMFACTORY->createString( item, value_string );
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_QNAME;
+ node_name = sent_qname_;
+ GENV_ITEMFACTORY->createAttributeNode(
+ attr_node, result, node_name, type_name, item
+ );
+
+ value_string = token->value();
+ GENV_ITEMFACTORY->createString( item, value_string );
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_QNAME;
+ node_name = value_qname_;
+ GENV_ITEMFACTORY->createAttributeNode(
+ attr_node, result, node_name, type_name, item
+ );
+
+ if ( store::Item const *const token_item = token->item() ) {
+ if ( GENV_STORE.getNodeReference( item, token_item ) ) {
+ item->getStringValue2( value_string );
+ GENV_ITEMFACTORY->createString( item, value_string );
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_QNAME;
+ node_name = ref_qname_;
+ GENV_ITEMFACTORY->createAttributeNode(
+ attr_node, result, node_name, type_name, item
+ );
+ }
+ }
+
+ STACK_PUSH( true, state );
+ } // while
+ }
+
+ STACK_END( state );
+}
+
+void TokenizeIterator::resetImpl( PlanState &plan_state ) const {
+ NaryBaseIterator<TokenizeIterator,TokenizeIteratorState>::
+ resetImpl( plan_state );
+ TokenizeIteratorState *const state =
+ StateTraitsImpl<TokenizeIteratorState>::getState(
+ plan_state, this->theStateOffset
+ );
+ state->doc_tokens_->reset();
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+bool TokenizerPropertiesIterator::nextImpl( store::Item_t &result,
+ PlanState &plan_state ) const {
+ store::Item_t element, item, junk, name;
+ zstring base_uri;
+ iso639_1::type lang;
+ Tokenizer::Numbers no;
+ store::NsBindings const ns_bindings;
+ static_context const *sctx;
+ Tokenizer::ptr tokenizer;
+ store::Item_t type_name;
+ Tokenizer::Properties props;
+ TokenizerProvider const *tokenizer_provider;
+ zstring value_string;
+
+ PlanIteratorState *state;
+ DEFAULT_STACK_INIT( PlanIteratorState, state, plan_state );
+
+ sctx = getStaticContext();
+ ZORBA_ASSERT( sctx );
+ lang = get_lang_from( sctx );
+
+ if ( theChildren.size() > 0 ) {
+ consumeNext( item, theChildren[0], plan_state );
+ lang = get_lang_from( item, loc );
+ }
+
+ tokenizer_provider = GENV_STORE.getTokenizerProvider();
+ ZORBA_ASSERT( tokenizer_provider );
+ if ( !tokenizer_provider->getTokenizer( lang, &no, &tokenizer ) )
+ throw XQUERY_EXCEPTION(
+ err::FTST0009,
+ ERROR_PARAMS(
+ iso639_1::string_of[ lang ], ZED( FTST0009_BadTokenizerLang )
+ )
+ );
+ tokenizer->properties( &props );
+
+ GENV_ITEMFACTORY->createQName(
+ name, static_context::ZORBA_FULL_TEXT_FN_NS, "", "tokenizer-properties"
+ );
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_QNAME;
+ GENV_ITEMFACTORY->createElementNode(
+ result, nullptr, name, type_name, false, false, ns_bindings, base_uri
+ );
+
+ // uri="..."
+ GENV_ITEMFACTORY->createQName( name, "", "", "uri" );
+ GENV_ITEMFACTORY->createAnyURI( item, props.uri );
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_ATOMIC_QNAME;
+ GENV_ITEMFACTORY->createAttributeNode( junk, result, name, type_name, item );
+
+ // <comments-separate-tokens value="..."/>
+ GENV_ITEMFACTORY->createQName(
+ name, static_context::ZORBA_FULL_TEXT_FN_NS, "", "comments-separate-tokens"
+ );
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_ATOMIC_QNAME;
+ GENV_ITEMFACTORY->createElementNode(
+ element, result, name, type_name, false, false, ns_bindings, base_uri
+ );
+ GENV_ITEMFACTORY->createQName( name, "", "", "value" );
+ GENV_ITEMFACTORY->createBoolean( item, props.comments_separate_tokens );
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_ATOMIC_QNAME;
+ GENV_ITEMFACTORY->createAttributeNode( junk, element, name, type_name, item );
+
+ // <elements-separate-tokens value="..."/>
+ GENV_ITEMFACTORY->createQName(
+ name, static_context::ZORBA_FULL_TEXT_FN_NS, "", "elements-separate-tokens"
+ );
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_ATOMIC_QNAME;
+ GENV_ITEMFACTORY->createElementNode(
+ element, result, name, type_name, false, false, ns_bindings, base_uri
+ );
+ GENV_ITEMFACTORY->createQName( name, "", "", "value" );
+ GENV_ITEMFACTORY->createBoolean( item, props.elements_separate_tokens );
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_ATOMIC_QNAME;
+ GENV_ITEMFACTORY->createAttributeNode( junk, element, name, type_name, item );
+
+ // <processing-instructions-separate-tokens value="..."/>
+ GENV_ITEMFACTORY->createQName(
+ name, static_context::ZORBA_FULL_TEXT_FN_NS, "",
+ "processing-instructions-separate-tokens"
+ );
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_ATOMIC_QNAME;
+ GENV_ITEMFACTORY->createElementNode(
+ element, result, name, type_name, false, false, ns_bindings, base_uri
+ );
+ GENV_ITEMFACTORY->createQName( name, "", "", "value" );
+ GENV_ITEMFACTORY->createBoolean( item, props.processing_instructions_separate_tokens );
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_ATOMIC_QNAME;
+ GENV_ITEMFACTORY->createAttributeNode( junk, element, name, type_name, item );
+
+ // <supported-languages>...</supported-languages>
+ GENV_ITEMFACTORY->createQName(
+ name, static_context::ZORBA_FULL_TEXT_FN_NS, "", "supported-languages"
+ );
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_ATOMIC_QNAME;
+ GENV_ITEMFACTORY->createElementNode(
+ element, result, name, type_name, false, false, ns_bindings, base_uri
+ );
+
+ // <lang>...</lang>
+ FOR_EACH( Tokenizer::Properties::languages_type, i, props.languages ) {
+ store::Item_t lang_element;
+ type_name = GENV_TYPESYSTEM.XS_UNTYPED_ATOMIC_QNAME;
+ GENV_ITEMFACTORY->createQName(
+ name, static_context::ZORBA_FULL_TEXT_FN_NS, "", "lang"
+ );
+ GENV_ITEMFACTORY->createElementNode(
+ lang_element, element, name, type_name, false, false, ns_bindings,
+ base_uri
+ );
+ value_string = iso639_1::string_of[ *i ];
+ GENV_ITEMFACTORY->createTextNode( junk, lang_element.getp(), value_string );
+ }
+
+#ifndef ZORBA_NO_XMLSCHEMA
+ sctx->validate( result, result, StaticContextConsts::strict_validation );
+#endif /* ZORBA_NO_XMLSCHEMA */
+
+ STACK_PUSH( true, state );
+ STACK_END( state );
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+struct TokenizeStringIteratorCallback : Tokenizer::Callback {
+ void token( char const*, size_type, iso639_1::type, size_type, size_type,
+ size_type, Item const* );
+
+ FTTokenSeqIterator::FTTokens tokens_;
+};
+
+void TokenizeStringIteratorCallback::
+token( char const *utf8_s, size_type utf8_len, iso639_1::type lang,
+ size_type token_no, size_type sent_no, size_type para_no,
+ Item const *item ) {
+ store::Item const *const store_item =
+ item ? Unmarshaller::getInternalItem( *item ) : nullptr;
+
+ FTToken const token(
+ utf8_s, utf8_len, token_no, sent_no, para_no, store_item, lang
+ );
+ tokens_.push_back( token );
+}
+
+bool TokenizeStringIterator::nextImpl( store::Item_t &result,
+ PlanState &plan_state ) const {
+ store::Item_t item;
+ iso639_1::type lang;
+ static_context const *sctx;
+ zstring value_string;
+
+ TokenizeStringIteratorState *state;
+ DEFAULT_STACK_INIT( TokenizeStringIteratorState, state, plan_state );
+
+ sctx = getStaticContext();
+ ZORBA_ASSERT( sctx );
+ lang = get_lang_from( sctx );
+
+ if ( consumeNext( item, theChildren[0], plan_state ) ) {
+ item->getStringValue2( value_string );
+ if ( theChildren.size() > 1 ) {
+ consumeNext( item, theChildren[1], plan_state );
+ lang = get_lang_from( item, loc );
+ }
+
+ { // local scope
+ TokenizerProvider const *const tokenizer_provider =
+ GENV_STORE.getTokenizerProvider();
+ ZORBA_ASSERT( tokenizer_provider );
+ Tokenizer::Numbers no;
+ Tokenizer::ptr tokenizer;
+ if ( !tokenizer_provider->getTokenizer( lang, &no, &tokenizer ) )
+ throw XQUERY_EXCEPTION(
+ err::FTST0009,
+ ERROR_PARAMS(
+ iso639_1::string_of[ lang ], ZED( FTST0009_BadTokenizerLang )
+ )
+ );
+
+ TokenizeStringIteratorCallback callback;
+ tokenizer->tokenize_string(
+ value_string.data(), value_string.size(), lang, false, callback
+ );
+ state->string_tokens_.take( callback.tokens_ );
+ } // local scope
+
+ while ( state->string_tokens_.hasNext() ) {
+ FTToken const *token;
+ token = state->string_tokens_.next();
+ ZORBA_ASSERT( token );
+ value_string = token->value();
+ GENV_ITEMFACTORY->createString( result, value_string );
+ STACK_PUSH( true, state );
+ }
+ }
+
+ STACK_END( state );
+}
+
+void TokenizeStringIterator::resetImpl( PlanState &plan_state ) const {
+ NaryBaseIterator<TokenizeStringIterator,TokenizeStringIteratorState>::
+ resetImpl( plan_state );
+ TokenizeStringIteratorState *const state =
+ StateTraitsImpl<TokenizeStringIteratorState>::getState(
+ plan_state, this->theStateOffset
+ );
+ state->string_tokens_.reset();
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+} // namespace zorba
+
+#endif /* ZORBA_NO_FULL_TEXT */
+
+/* vim:set et sw=2 ts=2: */
=== added file 'src/runtime/full_text/ft_module_impl.h'
--- src/runtime/full_text/ft_module_impl.h 1970-01-01 00:00:00 +0000
+++ src/runtime/full_text/ft_module_impl.h 2012-05-16 22:51:21 +0000
@@ -0,0 +1,32 @@
+/*
+ * Copyright 2006-2008 The FLWOR Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef ZORBA_FT_MODULE_IMPL_H
+#define ZORBA_FT_MODULE_IMPL_H
+
+namespace zorba {
+
+class static_context;
+
+///////////////////////////////////////////////////////////////////////////////
+
+void populate_context_ft_module_impl( static_context *sctx );
+
+///////////////////////////////////////////////////////////////////////////////
+
+} // namespace zorba
+#endif /* ZORBA_FT_MODULE_IMPL_H */
+/* vim:set et sw=2 ts=2: */
=== modified file 'src/runtime/full_text/ft_query_item.h'
--- src/runtime/full_text/ft_query_item.h 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/ft_query_item.h 2012-05-16 22:51:21 +0000
@@ -18,6 +18,7 @@
#define ZORBA_FULL_TEXT_FT_QUERY_ITEM_H
#include <list>
+#include <vector>
#include "store/api/ft_token_iterator.h"
@@ -59,7 +60,7 @@
void reset();
private:
- typedef std::list<Mark_t> MarkSeq;
+ typedef std::vector<Mark_t> MarkSeq;
struct LocalMark : Mark {
MarkSeq marks_;
=== modified file 'src/runtime/full_text/ft_single_token_iterator.h'
--- src/runtime/full_text/ft_single_token_iterator.h 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/ft_single_token_iterator.h 2012-05-16 22:51:21 +0000
@@ -17,8 +17,6 @@
#ifndef ZORBA_FULL_TEXT_SINGLE_TOKEN_ITERATOR_H
#define ZORBA_FULL_TEXT_SINGLE_TOKEN_ITERATOR_H
-#include <list>
-
#include "store/api/ft_token_iterator.h"
namespace zorba {
=== modified file 'src/runtime/full_text/ft_stop_words_set.cpp'
--- src/runtime/full_text/ft_stop_words_set.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/ft_stop_words_set.cpp 2012-05-16 22:51:21 +0000
@@ -17,14 +17,14 @@
#include <zorba/config.h>
-#include <util/ascii_util.h>
-#include <util/cxx_util.h>
-#include <util/mmap_file.h>
-#include <util/stl_util.h>
-#include <util/uri_util.h>
-#include <context/static_context.h>
-#include <context/uri_resolver.h>
-#include <zorbautils/locale.h>
+#include "context/static_context.h"
+#include "context/uri_resolver.h"
+#include "util/ascii_util.h"
+#include "util/cxx_util.h"
+#include "util/mmap_file.h"
+#include "util/stl_util.h"
+#include "util/uri_util.h"
+#include "zorbautils/locale.h"
#include "ft_stop_words_set.h"
@@ -57,7 +57,7 @@
case LANG(pt);
case LANG(sv);
default:
- return 0;
+ return nullptr;
}
}
@@ -72,10 +72,9 @@
///////////////////////////////////////////////////////////////////////////////
-void ft_stop_words_set::apply_word( zstring const &word, set_t &word_set,
+void ft_stop_words_set::apply_word( zstring const &word, word_set_t &word_set,
ft_stop_words_unex::type mode ) {
// TODO: should "word" be converted to lower-case?
- std::cout << "applying word " << word << std::endl;
switch ( mode ) {
case ft_stop_words_unex::union_:
word_set.insert( word );
@@ -87,33 +86,30 @@
}
void ft_stop_words_set::apply_word( char const *begin, char const *end,
- set_t &word_set,
+ word_set_t &word_set,
ft_stop_words_unex::type mode ) {
- set_t::value_type const word( begin, end - begin );
+ word_set_t::value_type const word( begin, end - begin );
apply_word( word, word_set, mode );
}
-ft_stop_words_set const*
+ft_stop_words_set::ptr
ft_stop_words_set::construct( ftstop_word_option const &option,
iso639_1::type lang,
static_context const& sctx ) {
bool must_delete = false;
- set_t *word_set = nullptr; // pointless init. to stifle warning
+ word_set_t *word_set = nullptr; // pointless init. to stifle warning
switch ( option.get_mode() ) {
case ft_stop_words_mode::with:
- word_set = new set_t;
+ word_set = new word_set_t;
must_delete = true;
break;
case ft_stop_words_mode::with_default:
- word_set = get_default_word_set_for( lang );
- if ( !word_set ) {
- // TODO: throw exception?
- return 0;
- }
- break;
+ if ( (word_set = get_default_word_set_for( lang )) )
+ break;
+ // no break;
case ft_stop_words_mode::without:
- return 0;
+ return ptr();
}
FOR_EACH( ftstop_word_option::list_t, ftsw, option.get_stop_words() ) {
@@ -122,31 +118,30 @@
if ( !uri.empty() ) {
if ( !must_delete ) {
- word_set = new set_t( *word_set );
+ word_set = new word_set_t( *word_set );
must_delete = true;
}
zstring error_msg;
std::auto_ptr<internal::Resource> rsrc =
- sctx.resolve_uri(uri, internal::EntityData::STOP_WORDS, error_msg);
- internal::StreamResource* stream_rsrc =
- dynamic_cast<internal::StreamResource*>(rsrc.get());
+ sctx.resolve_uri( uri, internal::EntityData::STOP_WORDS, error_msg );
+ internal::StreamResource *const stream_rsrc =
+ dynamic_cast<internal::StreamResource*>( rsrc.get() );
if ( !stream_rsrc ) {
// Technically this should be thrown during static analysis.
- throw ZORBA_EXCEPTION(err::FTST0008, ERROR_PARAMS(uri));
+ throw XQUERY_EXCEPTION( err::FTST0008, ERROR_PARAMS( uri ) );
}
- std::istream* stream = stream_rsrc->getStream();
+ std::istream *const stream = stream_rsrc->getStream();
bool in_word = false;
zstring cur_word;
- cur_word.reserve(128);
+ cur_word.reserve( 128 );
char c;
- while (stream->good()) {
- stream->get(c);
+ while ( stream->good() ) {
+ stream->get( c );
// Have to check for EOF *after* attempting the read
- if (stream->eof()) {
+ if ( stream->eof() )
break;
- }
if ( is_word_char( c ) ) {
if ( !in_word ) {
cur_word.clear();
@@ -167,25 +162,31 @@
ftstop_words::list_t const &word_list = (*ftsw)->get_list();
if ( !word_list.empty() ) {
if ( !must_delete ) {
- word_set = new set_t( *word_set );
+ word_set = new word_set_t( *word_set );
must_delete = true;
}
FOR_EACH( ftstop_words::list_t, word, word_list )
apply_word( *word, *word_set, mode );
}
}
- return new ft_stop_words_set( word_set, must_delete );
-}
-
-ft_stop_words_set::set_t*
+ return ptr( new ft_stop_words_set( word_set, must_delete ) );
+}
+
+ft_stop_words_set const*
+ft_stop_words_set::get_default( iso639_1::type lang ) {
+ word_set_t const *const word_set = get_default_word_set_for( lang );
+ return word_set ? new ft_stop_words_set( word_set, false ) : nullptr;
+}
+
+ft_stop_words_set::word_set_t*
ft_stop_words_set::get_default_word_set_for( iso639_1::type lang ) {
- static set_t* cached_word_sets[ iso639_1::NUM_ENTRIES ];
+ static word_set_t *cached_word_sets[ iso639_1::NUM_ENTRIES ];
if ( !lang )
lang = get_host_lang();
- set_t *&word_set = cached_word_sets[ lang ];
+ word_set_t *&word_set = cached_word_sets[ lang ];
if ( !word_set ) {
if ( ft_stop_table const table = get_table_for( lang ) ) {
- word_set = new set_t;
+ word_set = new word_set_t;
for ( ft_stop_table word = table; *word; ++word )
word_set->insert( *word );
}
=== modified file 'src/runtime/full_text/ft_stop_words_set.h'
--- src/runtime/full_text/ft_stop_words_set.h 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/ft_stop_words_set.h 2012-05-16 22:51:21 +0000
@@ -20,6 +20,7 @@
#include <set>
#include <zorba/locale.h>
+#include <zorba/internal/unique_ptr.h>
#include "compiler/expression/ftnode.h"
#include "zorbatypes/zstring.h"
@@ -27,26 +28,29 @@
namespace zorba {
/**
- * An %ft_stop_words_set is (as its name suggests) a set of stop-wors.
+ * An %ft_stop_words_set is (as its name suggests) a set of stop-words.
*/
class ft_stop_words_set {
public:
+ typedef std::unique_ptr<ft_stop_words_set const> ptr;
+
~ft_stop_words_set() {
if ( delete_ )
delete word_set_;
}
/**
- * Constructs an %ft_stop_words_set.
+ * Constructs an %ft_stop_words_set for the given language.
*
* @param option The ftstop_word_option to use to possibly add or remove
* stop-words.
* @param lang The language of the stop-words.
- * @return Returns a new %ft_stop_words_set.
+ * @return Returns a new %ft_stop_words_set or \c nullptr if stop-words for
+ * \a lang are unsupported.
*/
- static ft_stop_words_set const* construct( ftstop_word_option const &option,
- locale::iso639_1::type lang,
- static_context const& sctx );
+ static ptr construct( ftstop_word_option const &option,
+ locale::iso639_1::type lang,
+ static_context const& sctx );
/**
* Checks whether this %ft_stop_words_set contains the given word.
@@ -60,22 +64,33 @@
return word_set_->find( word ) != word_set_->end();
}
+ /**
+ * Gets the default %ft_stop_words_set.
+ *
+ * @param lang The language of the stop-words.
+ * @return Returns said default or \c nullptr if stop-words for \a lang are
+ * unsupported.
+ */
+ static ft_stop_words_set const* get_default( locale::iso639_1::type lang );
+
private:
- typedef std::set<zstring> set_t;
+ typedef std::set<zstring> word_set_t;
- set_t const *const word_set_;
+ word_set_t const *const word_set_;
bool const delete_;
- ft_stop_words_set( set_t const *word_set, bool must_delete ) :
+ ft_stop_words_set( word_set_t const *word_set, bool must_delete ) :
word_set_( word_set ), delete_( must_delete )
{
}
- static void apply_word( zstring const&, set_t&, ft_stop_words_unex::type );
- static void apply_word( char const*, char const*, set_t&,
- ft_stop_words_unex::type );
-
- static set_t* get_default_word_set_for( locale::iso639_1::type );
+ static void apply_word( zstring const&, word_set_t&,
+ ft_stop_words_unex::type );
+
+ static void apply_word( char const*, char const*, word_set_t&,
+ ft_stop_words_unex::type );
+
+ static word_set_t* get_default_word_set_for( locale::iso639_1::type );
// forbid these
ft_stop_words_set( ft_stop_words_set const& );
=== modified file 'src/runtime/full_text/ft_token_matcher.cpp'
--- src/runtime/full_text/ft_token_matcher.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/ft_token_matcher.cpp 2012-05-16 22:51:21 +0000
@@ -47,11 +47,16 @@
return false;
}
-inline ft_stop_words_set const* get_stop_words( ftmatch_options const &options,
- iso639_1::type lang,
- static_context const& sctx ) {
+//
+// This returns a raw pointer to work around a bug where ~ft_stop_words_set()
+// is being called when it shouldn't be due to ~unique_ptr() being called
+// without transferring ownership properly.
+//
+inline ft_stop_words_set const*
+get_stop_words( ftmatch_options const &options, iso639_1::type lang,
+ static_context const& sctx ) {
if ( ftstop_word_option const *const sw = options.get_stop_word_option() )
- return ft_stop_words_set::construct( *sw, lang, sctx );
+ return ft_stop_words_set::construct( *sw, lang, sctx ).release();
return nullptr;
}
@@ -69,7 +74,7 @@
}
ft_token_matcher::~ft_token_matcher() {
- delete stop_words_;
+ // out-of-line since it's virtual
}
///////////////////////////////////////////////////////////////////////////////
@@ -83,8 +88,8 @@
void ft_token_matcher::match_stemmer::
operator()( string_t const &word, iso639_1::type lang,
string_t *result ) const {
- internal::Stemmer::ptr stemmer( provider_->get_stemmer( lang ) );
- if ( stemmer )
+ internal::Stemmer::ptr stemmer;
+ if ( provider_->getStemmer( lang, &stemmer ) )
stemmer->stem( word, lang, result );
else
*result = word;
=== modified file 'src/runtime/full_text/ft_token_matcher.h'
--- src/runtime/full_text/ft_token_matcher.h 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/ft_token_matcher.h 2012-05-16 22:51:21 +0000
@@ -62,7 +62,7 @@
locale::iso639_1::type const lang_;
bool const stemming_;
match_stemmer const stemmer_;
- ft_stop_words_set const *const stop_words_;
+ ft_stop_words_set::ptr stop_words_;
bool const wildcards_;
};
=== modified file 'src/runtime/full_text/ft_token_seq_iterator.cpp'
--- src/runtime/full_text/ft_token_seq_iterator.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/ft_token_seq_iterator.cpp 2012-05-16 22:51:21 +0000
@@ -25,8 +25,7 @@
namespace zorba {
FTTokenSeqIterator::FTTokenSeqIterator( FTTokens &tokens ) {
- tokens_.swap( tokens );
- pos_ = 0;
+ take( tokens );
}
FTTokenSeqIterator::~FTTokenSeqIterator() {
@@ -38,7 +37,7 @@
}
FTTokenIterator::index_t FTTokenSeqIterator::end() const {
- return (FTTokenIterator::index_t)tokens_.size();;
+ return static_cast<FTTokenIterator::index_t>( tokens_.size() );
}
bool FTTokenSeqIterator::hasNext() const {
@@ -61,5 +60,10 @@
pos_ = 0;
}
+void FTTokenSeqIterator::take( FTTokens &tokens ) {
+ tokens.swap( tokens_ );
+ pos_ = 0;
+}
+
} // namespace zorba
/* vim:set et sw=2 ts=2: */
=== modified file 'src/runtime/full_text/ft_token_seq_iterator.h'
--- src/runtime/full_text/ft_token_seq_iterator.h 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/ft_token_seq_iterator.h 2012-05-16 22:51:21 +0000
@@ -33,9 +33,12 @@
public:
typedef std::vector<FTToken> FTTokens;
+ FTTokenSeqIterator() { }
FTTokenSeqIterator( FTTokens& );
~FTTokenSeqIterator();
+ void take( FTTokens& );
+
// inherited
index_t begin() const;
index_t end() const;
=== modified file 'src/runtime/full_text/ft_token_span.h'
--- src/runtime/full_text/ft_token_span.h 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/ft_token_span.h 2012-05-16 22:51:21 +0000
@@ -20,7 +20,7 @@
#ifndef NDEBUG
#include <iostream>
#endif /* NDEBUG */
-#include <list>
+#include <vector>
#include "zorbatypes/ft_token.h"
@@ -51,7 +51,7 @@
/**
* An %ft_token_spans contains zero or more ft_token_span objects.
*/
-typedef std::list<ft_token_span> ft_token_spans;
+typedef std::vector<ft_token_span> ft_token_spans;
////////// Comparison operators ///////////////////////////////////////////////
=== added file 'src/runtime/full_text/ft_util.cpp'
--- src/runtime/full_text/ft_util.cpp 1970-01-01 00:00:00 +0000
+++ src/runtime/full_text/ft_util.cpp 2012-05-16 22:51:21 +0000
@@ -0,0 +1,45 @@
+/*
+ * Copyright 2006-2008 The FLWOR Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "stdafx.h"
+
+#include <stdexcept>
+
+#include "diagnostics/xquery_diagnostics.h"
+#include "zorbatypes/numconversions.h"
+
+#include "ft_util.h"
+
+namespace zorba {
+
+///////////////////////////////////////////////////////////////////////////////
+
+ft_int to_ft_int( xs_integer const &i ) {
+ try {
+ return to_xs_unsignedInt( i );
+ }
+ catch ( std::invalid_argument const& ) { // for negative numbers
+ throw XQUERY_EXCEPTION( err::FOCA0003, ERROR_PARAMS( i.toString() ) );
+ }
+ catch ( std::range_error const& ) {
+ throw XQUERY_EXCEPTION( err::FOCA0003, ERROR_PARAMS( i.toString() ) );
+ }
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+} // namespace zorba
+/* vim:set et sw=2 ts=2: */
=== modified file 'src/runtime/full_text/ft_util.h'
--- src/runtime/full_text/ft_util.h 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/ft_util.h 2012-05-16 22:51:21 +0000
@@ -21,6 +21,7 @@
#include "compiler/expression/ftnode.h"
#include "zorbatypes/schema_types.h"
+#include "util/cxx_util.h"
#include "ft_match.h"
@@ -70,7 +71,7 @@
if ( ftthesaurus_option const *const t = options->get_thesaurus_option() )
if ( !t->no_thesaurus() )
return t;
- return 0;
+ return nullptr;
}
/**
@@ -87,6 +88,16 @@
return false;
}
+/**
+ * Attempts to convert an \c xs:integer to an \c xs:unsignedInt.
+ *
+ * @param i The \c xs:integer to convert.
+ * @return Returns the value converted to an \c xs:unsignedInt.
+ * @throws \c err::FOCA0003 if the value can not be represented as an \c
+ * xs:unsignedInt.
+ */
+ft_int to_ft_int( xs_integer const &i );
+
} // namespace zorba
#endif /* ZORBA_FULL_TEXT_UTIL_H */
/* vim:set et sw=2 ts=2: */
=== modified file 'src/runtime/full_text/ftcontains_visitor.cpp'
--- src/runtime/full_text/ftcontains_visitor.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/ftcontains_visitor.cpp 2012-05-16 22:51:21 +0000
@@ -27,7 +27,6 @@
#include "util/cxx_util.h"
#include "util/indent.h"
#include "util/stl_util.h"
-#include "zorbatypes/numconversions.h"
#ifndef NDEBUG
#include "system/properties.h"
@@ -77,15 +76,6 @@
return d.getNumber();
}
-inline ft_int to_ft_int( xs_integer const &i ) {
- try {
- return to_xs_unsignedInt( i );
- }
- catch ( std::range_error const& ) {
- throw XQUERY_EXCEPTION( err::FOCA0003, ERROR_PARAMS( i.toString() ) );
- }
-}
-
////////// PUSH/POP ///////////////////////////////////////////////////////////
/**
=== modified file 'src/runtime/full_text/full_text.h'
--- src/runtime/full_text/full_text.h 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/full_text.h 2012-05-16 22:51:21 +0000
@@ -40,7 +40,7 @@
SERIALIZABLE_CLASS_CONSTRUCTOR2(FTContainsIterator,base_type);
void serialize( serialization::Archiver& );
- typedef std::list<PlanIter_t> sub_iter_list_t;
+ typedef std::vector<PlanIter_t> sub_iter_list_t;
FTContainsIterator(
static_context*,
=== modified file 'src/runtime/full_text/icu_tokenizer.cpp'
--- src/runtime/full_text/icu_tokenizer.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/icu_tokenizer.cpp 2012-05-16 22:51:21 +0000
@@ -16,6 +16,7 @@
#include "stdafx.h"
#include <cctype>
+#include <cstring>
#include <unicode/unistr.h>
#define DEBUG_TOKENIZER 0
@@ -54,6 +55,8 @@
public:
typedef Tokenizer::size_type size_type;
+ temp_token( iso639_1::type lang ) : lang_( lang ) { }
+
void append( char const *s, size_type slen ) {
value_.append( s, slen );
}
@@ -66,12 +69,14 @@
return value_.empty();
}
- void send( void *payload, Tokenizer::Callback &callback ) {
+ void send( Item const *item, Tokenizer::Callback &callback ) {
if ( !empty() ) {
# if DEBUG_TOKENIZER
cout << "TOKEN: \"" << value_ << "\" (" << pos_ << ',' << sent_ << ',' << para_ << ")\n";
# endif
- callback( value_.data(), value_.size(), pos_, sent_, para_, payload );
+ callback.token(
+ value_.data(), value_.size(), lang_, pos_, sent_, para_, item
+ );
clear();
}
}
@@ -87,6 +92,7 @@
private:
string value_;
+ iso639_1::type const lang_;
size_type pos_, sent_, para_;
};
@@ -158,6 +164,21 @@
delete this;
}
+void ICU_Tokenizer::properties( Properties *p ) const {
+ p->comments_separate_tokens = true;
+ p->elements_separate_tokens = true;
+ p->processing_instructions_separate_tokens = true;
+
+ p->languages.clear();
+ for ( int32_t n = ubrk_countAvailable(), i = 0; i < n; ++i ) {
+ if ( char const *const icu_locale = ubrk_getAvailable( i ) )
+ if ( iso639_1::type const lang = find_lang( icu_locale ) )
+ p->languages.push_back( lang );
+ }
+
+ p->uri = "http://www.zorba-xquery.com/full-text/tokenizer/icu";;
+}
+
#define HANDLE_BACKSLASH() \
if ( !got_backslash ) ; else { \
got_backslash = in_wild = false; \
@@ -174,9 +195,9 @@
#define IS_WORD_BREAK(TYPE,STATUS) \
( (STATUS) >= UBRK_WORD_##TYPE && (STATUS) < UBRK_WORD_##TYPE##_LIMIT )
-void ICU_Tokenizer::tokenize( char const *utf8_s, size_type utf8_len,
- iso639_1::type lang, bool wildcards,
- Callback &callback, void *payload ) {
+void ICU_Tokenizer::tokenize_string( char const *utf8_s, size_type utf8_len,
+ iso639_1::type lang, bool wildcards,
+ Callback &callback, Item const *item ) {
ZORBA_ASSERT( lang == lang_ );
unicode::char_type *utf16_buf;
@@ -206,7 +227,7 @@
sent_it_->setText( utf16_s );
unicode::size_type sent_end = sent_it_->first(); sent_end = sent_it_->next();
- temp_token t;
+ temp_token t( lang );
// True only if the previous token was a backslash ('\').
bool got_backslash = false;
@@ -295,8 +316,8 @@
else if ( IS_WORD_BREAK( NUMBER, rule_status ) ) {
//
// "NUMBER" tokens are obviously for numbers. Note that a sequence of
- // digits containing a ',' (e.g., "1,2") is considered a single token by
- // ICU.
+ // digits containing either a '.' (e.g., "98.6") or a ',' (e.g., "1,2")
+ // are considered a single tokens by ICU.
//
# if DEBUG_TOKENIZER
cout << "(NUMBER)" << endl;
@@ -346,7 +367,7 @@
}
if ( !in_wild && !got_backslash )
- t.send( payload, callback );
+ t.send( item, callback );
set_token:
# if DEBUG_TOKENIZER
@@ -395,7 +416,7 @@
throw XQUERY_EXCEPTION(
err::FTDY0020, ERROR_PARAMS( "", ZED( UnbalancedChar_3 ), '}' )
);
- t.send( payload, callback );
+ t.send( item, callback );
// Incrementing "sent" here fixes:
// https://bugs.launchpad.net/bugs/897800
++numbers().sent;
@@ -406,10 +427,18 @@
///////////////////////////////////////////////////////////////////////////////
-Tokenizer::ptr
-ICU_TokenizerProvider::getTokenizer( iso639_1::type lang,
- Tokenizer::Numbers &no ) const {
- return Tokenizer::ptr( new ICU_Tokenizer( lang, no ) );
+bool ICU_TokenizerProvider::getTokenizer( iso639_1::type lang,
+ Tokenizer::Numbers *num,
+ Tokenizer::ptr *t ) const {
+ for ( int32_t n = ubrk_countAvailable(), i = 0; i < n; ++i ) {
+ if ( char const *const icu_locale = ubrk_getAvailable( i ) )
+ if ( lang == find_lang( icu_locale ) ) {
+ if ( num && t )
+ t->reset( new ICU_Tokenizer( lang, *num ) );
+ return true;
+ }
+ }
+ return false;
}
///////////////////////////////////////////////////////////////////////////////
=== modified file 'src/runtime/full_text/icu_tokenizer.h'
--- src/runtime/full_text/icu_tokenizer.h 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/icu_tokenizer.h 2012-05-16 22:51:21 +0000
@@ -48,8 +48,9 @@
// inherited
void destroy() const;
- void tokenize( char const*, size_type, locale::iso639_1::type, bool,
- Callback&, void* );
+ void properties( Properties* ) const;
+ void tokenize_string( char const*, size_type, locale::iso639_1::type, bool,
+ Callback&, Item const* );
private:
typedef std::unique_ptr<RuleBasedBreakIterator> rbbi_ptr;
@@ -63,10 +64,11 @@
class ICU_TokenizerProvider : public TokenizerProvider {
public:
- ICU_TokenizerProvider () {}
+ ICU_TokenizerProvider() { } // needed to work-around compiler bug
+
// inherited
- Tokenizer::ptr
- getTokenizer( locale::iso639_1::type, Tokenizer::Numbers& ) const;
+ bool getTokenizer( locale::iso639_1::type, Tokenizer::Numbers* = 0,
+ Tokenizer::ptr* = 0 ) const;
};
///////////////////////////////////////////////////////////////////////////////
=== modified file 'src/runtime/full_text/latin_tokenizer.cpp'
--- src/runtime/full_text/latin_tokenizer.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/latin_tokenizer.cpp 2012-05-16 22:51:21 +0000
@@ -82,15 +82,26 @@
return ++s < end ? *s : '\0';
}
+void LatinTokenizer::properties( Properties *p ) const {
+ p->comments_separate_tokens = true;
+ p->elements_separate_tokens = true;
+ p->processing_instructions_separate_tokens = true;
+
+ p->languages.clear();
+ p->languages.push_back( iso639_1::en );
+
+ p->uri = "http://www.zorba-xquery.com/full-text/tokenizer/latin";;
+}
+
#define HANDLE_BACKSLASH() \
if ( !got_backslash ) ; else { \
got_backslash = in_wild = false; \
break; \
}
-void LatinTokenizer::tokenize( char const *s, size_type s_len,
- iso639_1::type lang, bool wildcards,
- Callback &callback, void *payload ) {
+void LatinTokenizer::tokenize_string( char const *s, size_type s_len,
+ iso639_1::type lang, bool wildcards,
+ Callback &callback, Item const *item ) {
bool got_backslash = false;
bool in_wild = false;
string_type token;
@@ -167,7 +178,7 @@
} else {
if ( is_word_char( *s ) )
token += *s;
- else if ( send_token( token, callback, payload ) ) {
+ else if ( send_token( token, lang, callback, item ) ) {
token.clear();
t_type_ = t_generic;
}
@@ -203,13 +214,13 @@
}
} // for
- send_token( token, callback, payload );
+ send_token( token, lang, callback, item );
}
#define PRINT_TOKENS 0
-bool LatinTokenizer::send_token( string_type const &token,
- Callback &callback, void *payload ) {
+bool LatinTokenizer::send_token( string_type const &token, iso639_1::type lang,
+ Callback &callback, Item const *item ) {
if ( !token.empty() ) {
#if PRINT_TOKENS
cout << "t=" << setw(2) << numbers().token
@@ -219,8 +230,8 @@
#endif /* PRINT_TOKENS */
callback(
- token.data(), token.size(),
- numbers().token, numbers().sent, numbers().para, payload
+ token.data(), token.size(), lang,
+ numbers().token, numbers().sent, numbers().para, item
);
++numbers().token;
return true;
@@ -230,10 +241,17 @@
///////////////////////////////////////////////////////////////////////////////
-Tokenizer::ptr
-LatinTokenizerProvider::getTokenizer( iso639_1::type lang,
- Tokenizer::Numbers &num ) const {
- return Tokenizer::ptr( new LatinTokenizer( num ) );
+bool LatinTokenizerProvider::getTokenizer( iso639_1::type lang,
+ Tokenizer::Numbers *num,
+ Tokenizer::ptr *t ) const {
+ switch ( lang ) {
+ case iso639_1::en:
+ if ( num && t )
+ t->reset( new LatinTokenizer( *num ) );
+ return true;
+ default:
+ return false;
+ }
}
///////////////////////////////////////////////////////////////////////////////
=== modified file 'src/runtime/full_text/latin_tokenizer.h'
--- src/runtime/full_text/latin_tokenizer.h 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/latin_tokenizer.h 2012-05-16 22:51:21 +0000
@@ -38,8 +38,9 @@
// inherited
void destroy() const;
- void tokenize( char const*, size_type, locale::iso639_1::type, bool,
- Callback&, void* );
+ void properties( Properties* ) const;
+ void tokenize_string( char const*, size_type, iso639_1::type, bool, Callback&,
+ Item const* );
private:
typedef zstring string_type;
@@ -56,7 +57,8 @@
static bool is_word_begin_char( char );
bool is_word_char( char );
static char peek( char const *s, char const *end );
- bool send_token( string_type const &token, Callback&, void* );
+ bool send_token( string_type const &token, locale::iso639_1::type, Callback&,
+ Item const* );
};
///////////////////////////////////////////////////////////////////////////////
@@ -64,8 +66,8 @@
class LatinTokenizerProvider : public TokenizerProvider {
public:
// inherited
- Tokenizer::ptr getTokenizer( locale::iso639_1::type,
- Tokenizer::Numbers& ) const;
+ bool getTokenizer( locale::iso639_1::type, Tokenizer::Numbers* = 0,
+ Tokenizer::ptr* = 0 ) const;
};
///////////////////////////////////////////////////////////////////////////////
=== added directory 'src/runtime/full_text/pregenerated'
=== added file 'src/runtime/full_text/pregenerated/ft_module.cpp'
--- src/runtime/full_text/pregenerated/ft_module.cpp 1970-01-01 00:00:00 +0000
+++ src/runtime/full_text/pregenerated/ft_module.cpp 2012-05-16 22:51:21 +0000
@@ -0,0 +1,362 @@
+/*
+ * Copyright 2006-2008 The FLWOR Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// ******************************************
+// * *
+// * THIS IS A GENERATED FILE. DO NOT EDIT! *
+// * SEE .xml FILE WITH SAME NAME *
+// * *
+// ******************************************
+
+#include "stdafx.h"
+#include "zorbatypes/rchandle.h"
+#include "zorbatypes/zstring.h"
+#include "runtime/visitors/planiter_visitor.h"
+#include "runtime/full_text/ft_module.h"
+#include "system/globalenv.h"
+
+
+#include "store/api/iterator.h"
+
+namespace zorba {
+
+#ifndef ZORBA_NO_FULL_TEXT
+// <CurrentLangIterator>
+CurrentLangIterator::class_factory<CurrentLangIterator>
+CurrentLangIterator::g_class_factory;
+
+
+void CurrentLangIterator::accept(PlanIterVisitor& v) const {
+ v.beginVisit(*this);
+
+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
+ for ( ; lIter != lEnd; ++lIter ){
+ (*lIter)->accept(v);
+ }
+
+ v.endVisit(*this);
+}
+
+CurrentLangIterator::~CurrentLangIterator() {}
+
+// </CurrentLangIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <HostLangIterator>
+HostLangIterator::class_factory<HostLangIterator>
+HostLangIterator::g_class_factory;
+
+
+void HostLangIterator::accept(PlanIterVisitor& v) const {
+ v.beginVisit(*this);
+
+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
+ for ( ; lIter != lEnd; ++lIter ){
+ (*lIter)->accept(v);
+ }
+
+ v.endVisit(*this);
+}
+
+HostLangIterator::~HostLangIterator() {}
+
+// </HostLangIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <IsStemLangSupportedIterator>
+IsStemLangSupportedIterator::class_factory<IsStemLangSupportedIterator>
+IsStemLangSupportedIterator::g_class_factory;
+
+
+void IsStemLangSupportedIterator::accept(PlanIterVisitor& v) const {
+ v.beginVisit(*this);
+
+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
+ for ( ; lIter != lEnd; ++lIter ){
+ (*lIter)->accept(v);
+ }
+
+ v.endVisit(*this);
+}
+
+IsStemLangSupportedIterator::~IsStemLangSupportedIterator() {}
+
+// </IsStemLangSupportedIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <IsStopWordIterator>
+IsStopWordIterator::class_factory<IsStopWordIterator>
+IsStopWordIterator::g_class_factory;
+
+
+void IsStopWordIterator::accept(PlanIterVisitor& v) const {
+ v.beginVisit(*this);
+
+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
+ for ( ; lIter != lEnd; ++lIter ){
+ (*lIter)->accept(v);
+ }
+
+ v.endVisit(*this);
+}
+
+IsStopWordIterator::~IsStopWordIterator() {}
+
+// </IsStopWordIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <IsStopWordLangSupportedIterator>
+IsStopWordLangSupportedIterator::class_factory<IsStopWordLangSupportedIterator>
+IsStopWordLangSupportedIterator::g_class_factory;
+
+
+void IsStopWordLangSupportedIterator::accept(PlanIterVisitor& v) const {
+ v.beginVisit(*this);
+
+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
+ for ( ; lIter != lEnd; ++lIter ){
+ (*lIter)->accept(v);
+ }
+
+ v.endVisit(*this);
+}
+
+IsStopWordLangSupportedIterator::~IsStopWordLangSupportedIterator() {}
+
+// </IsStopWordLangSupportedIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <IsThesaurusLangSupportedIterator>
+IsThesaurusLangSupportedIterator::class_factory<IsThesaurusLangSupportedIterator>
+IsThesaurusLangSupportedIterator::g_class_factory;
+
+
+void IsThesaurusLangSupportedIterator::accept(PlanIterVisitor& v) const {
+ v.beginVisit(*this);
+
+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
+ for ( ; lIter != lEnd; ++lIter ){
+ (*lIter)->accept(v);
+ }
+
+ v.endVisit(*this);
+}
+
+IsThesaurusLangSupportedIterator::~IsThesaurusLangSupportedIterator() {}
+
+// </IsThesaurusLangSupportedIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <IsTokenizerLangSupportedIterator>
+IsTokenizerLangSupportedIterator::class_factory<IsTokenizerLangSupportedIterator>
+IsTokenizerLangSupportedIterator::g_class_factory;
+
+
+void IsTokenizerLangSupportedIterator::accept(PlanIterVisitor& v) const {
+ v.beginVisit(*this);
+
+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
+ for ( ; lIter != lEnd; ++lIter ){
+ (*lIter)->accept(v);
+ }
+
+ v.endVisit(*this);
+}
+
+IsTokenizerLangSupportedIterator::~IsTokenizerLangSupportedIterator() {}
+
+// </IsTokenizerLangSupportedIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <StemIterator>
+StemIterator::class_factory<StemIterator>
+StemIterator::g_class_factory;
+
+
+void StemIterator::accept(PlanIterVisitor& v) const {
+ v.beginVisit(*this);
+
+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
+ for ( ; lIter != lEnd; ++lIter ){
+ (*lIter)->accept(v);
+ }
+
+ v.endVisit(*this);
+}
+
+StemIterator::~StemIterator() {}
+
+// </StemIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <StripDiacriticsIterator>
+StripDiacriticsIterator::class_factory<StripDiacriticsIterator>
+StripDiacriticsIterator::g_class_factory;
+
+
+void StripDiacriticsIterator::accept(PlanIterVisitor& v) const {
+ v.beginVisit(*this);
+
+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
+ for ( ; lIter != lEnd; ++lIter ){
+ (*lIter)->accept(v);
+ }
+
+ v.endVisit(*this);
+}
+
+StripDiacriticsIterator::~StripDiacriticsIterator() {}
+
+// </StripDiacriticsIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <ThesaurusLookupIterator>
+ThesaurusLookupIterator::class_factory<ThesaurusLookupIterator>
+ThesaurusLookupIterator::g_class_factory;
+
+
+void ThesaurusLookupIterator::accept(PlanIterVisitor& v) const {
+ v.beginVisit(*this);
+
+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
+ for ( ; lIter != lEnd; ++lIter ){
+ (*lIter)->accept(v);
+ }
+
+ v.endVisit(*this);
+}
+
+ThesaurusLookupIterator::~ThesaurusLookupIterator() {}
+
+ThesaurusLookupIteratorState::ThesaurusLookupIteratorState() {}
+
+ThesaurusLookupIteratorState::~ThesaurusLookupIteratorState() {}
+
+
+void ThesaurusLookupIteratorState::reset(PlanState& planState) {
+ PlanIteratorState::reset(planState);
+}
+// </ThesaurusLookupIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <TokenizeIterator>
+TokenizeIterator::class_factory<TokenizeIterator>
+TokenizeIterator::g_class_factory;
+
+
+void TokenizeIterator::accept(PlanIterVisitor& v) const {
+ v.beginVisit(*this);
+
+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
+ for ( ; lIter != lEnd; ++lIter ){
+ (*lIter)->accept(v);
+ }
+
+ v.endVisit(*this);
+}
+
+TokenizeIterator::~TokenizeIterator() {}
+
+TokenizeIteratorState::TokenizeIteratorState() {}
+
+TokenizeIteratorState::~TokenizeIteratorState() {}
+
+
+void TokenizeIteratorState::reset(PlanState& planState) {
+ PlanIteratorState::reset(planState);
+}
+// </TokenizeIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <TokenizerPropertiesIterator>
+TokenizerPropertiesIterator::class_factory<TokenizerPropertiesIterator>
+TokenizerPropertiesIterator::g_class_factory;
+
+
+void TokenizerPropertiesIterator::accept(PlanIterVisitor& v) const {
+ v.beginVisit(*this);
+
+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
+ for ( ; lIter != lEnd; ++lIter ){
+ (*lIter)->accept(v);
+ }
+
+ v.endVisit(*this);
+}
+
+TokenizerPropertiesIterator::~TokenizerPropertiesIterator() {}
+
+// </TokenizerPropertiesIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <TokenizeStringIterator>
+TokenizeStringIterator::class_factory<TokenizeStringIterator>
+TokenizeStringIterator::g_class_factory;
+
+
+void TokenizeStringIterator::accept(PlanIterVisitor& v) const {
+ v.beginVisit(*this);
+
+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
+ for ( ; lIter != lEnd; ++lIter ){
+ (*lIter)->accept(v);
+ }
+
+ v.endVisit(*this);
+}
+
+TokenizeStringIterator::~TokenizeStringIterator() {}
+
+TokenizeStringIteratorState::TokenizeStringIteratorState() {}
+
+TokenizeStringIteratorState::~TokenizeStringIteratorState() {}
+
+
+void TokenizeStringIteratorState::reset(PlanState& planState) {
+ PlanIteratorState::reset(planState);
+}
+// </TokenizeStringIterator>
+
+#endif
+
+}
+
+
=== added file 'src/runtime/full_text/pregenerated/ft_module.h'
--- src/runtime/full_text/pregenerated/ft_module.h 1970-01-01 00:00:00 +0000
+++ src/runtime/full_text/pregenerated/ft_module.h 2012-05-16 22:51:21 +0000
@@ -0,0 +1,563 @@
+/*
+ * Copyright 2006-2008 The FLWOR Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// ******************************************
+// * *
+// * THIS IS A GENERATED FILE. DO NOT EDIT! *
+// * SEE .xml FILE WITH SAME NAME *
+// * *
+// ******************************************
+#ifndef ZORBA_RUNTIME_FULL_TEXT_FT_MODULE_H
+#define ZORBA_RUNTIME_FULL_TEXT_FT_MODULE_H
+
+
+#include "common/shared_types.h"
+
+
+
+#include "runtime/base/narybase.h"
+#include "runtime/full_text/ft_token_seq_iterator.h"
+#include "runtime/full_text/thesaurus.h"
+
+
+namespace zorba {
+
+#ifndef ZORBA_NO_FULL_TEXT
+/**
+ *
+ * Author:
+ */
+class CurrentLangIterator : public NaryBaseIterator<CurrentLangIterator, PlanIteratorState>
+{
+public:
+ SERIALIZABLE_CLASS(CurrentLangIterator);
+
+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(CurrentLangIterator,
+ NaryBaseIterator<CurrentLangIterator, PlanIteratorState>);
+
+ void serialize( ::zorba::serialization::Archiver& ar)
+ {
+ serialize_baseclass(ar,
+ (NaryBaseIterator<CurrentLangIterator, PlanIteratorState>*)this);
+ }
+
+ CurrentLangIterator(
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& children)
+ :
+ NaryBaseIterator<CurrentLangIterator, PlanIteratorState>(sctx, loc, children)
+ {}
+
+ virtual ~CurrentLangIterator();
+
+ void accept(PlanIterVisitor& v) const;
+
+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
+};
+
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+/**
+ *
+ * Author:
+ */
+class HostLangIterator : public NaryBaseIterator<HostLangIterator, PlanIteratorState>
+{
+public:
+ SERIALIZABLE_CLASS(HostLangIterator);
+
+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(HostLangIterator,
+ NaryBaseIterator<HostLangIterator, PlanIteratorState>);
+
+ void serialize( ::zorba::serialization::Archiver& ar)
+ {
+ serialize_baseclass(ar,
+ (NaryBaseIterator<HostLangIterator, PlanIteratorState>*)this);
+ }
+
+ HostLangIterator(
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& children)
+ :
+ NaryBaseIterator<HostLangIterator, PlanIteratorState>(sctx, loc, children)
+ {}
+
+ virtual ~HostLangIterator();
+
+ void accept(PlanIterVisitor& v) const;
+
+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
+};
+
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+/**
+ *
+ * Author:
+ */
+class IsStemLangSupportedIterator : public NaryBaseIterator<IsStemLangSupportedIterator, PlanIteratorState>
+{
+public:
+ SERIALIZABLE_CLASS(IsStemLangSupportedIterator);
+
+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(IsStemLangSupportedIterator,
+ NaryBaseIterator<IsStemLangSupportedIterator, PlanIteratorState>);
+
+ void serialize( ::zorba::serialization::Archiver& ar)
+ {
+ serialize_baseclass(ar,
+ (NaryBaseIterator<IsStemLangSupportedIterator, PlanIteratorState>*)this);
+ }
+
+ IsStemLangSupportedIterator(
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& children)
+ :
+ NaryBaseIterator<IsStemLangSupportedIterator, PlanIteratorState>(sctx, loc, children)
+ {}
+
+ virtual ~IsStemLangSupportedIterator();
+
+ void accept(PlanIterVisitor& v) const;
+
+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
+};
+
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+/**
+ *
+ * Author:
+ */
+class IsStopWordIterator : public NaryBaseIterator<IsStopWordIterator, PlanIteratorState>
+{
+public:
+ SERIALIZABLE_CLASS(IsStopWordIterator);
+
+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(IsStopWordIterator,
+ NaryBaseIterator<IsStopWordIterator, PlanIteratorState>);
+
+ void serialize( ::zorba::serialization::Archiver& ar)
+ {
+ serialize_baseclass(ar,
+ (NaryBaseIterator<IsStopWordIterator, PlanIteratorState>*)this);
+ }
+
+ IsStopWordIterator(
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& children)
+ :
+ NaryBaseIterator<IsStopWordIterator, PlanIteratorState>(sctx, loc, children)
+ {}
+
+ virtual ~IsStopWordIterator();
+
+ void accept(PlanIterVisitor& v) const;
+
+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
+};
+
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+/**
+ *
+ * Author:
+ */
+class IsStopWordLangSupportedIterator : public NaryBaseIterator<IsStopWordLangSupportedIterator, PlanIteratorState>
+{
+public:
+ SERIALIZABLE_CLASS(IsStopWordLangSupportedIterator);
+
+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(IsStopWordLangSupportedIterator,
+ NaryBaseIterator<IsStopWordLangSupportedIterator, PlanIteratorState>);
+
+ void serialize( ::zorba::serialization::Archiver& ar)
+ {
+ serialize_baseclass(ar,
+ (NaryBaseIterator<IsStopWordLangSupportedIterator, PlanIteratorState>*)this);
+ }
+
+ IsStopWordLangSupportedIterator(
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& children)
+ :
+ NaryBaseIterator<IsStopWordLangSupportedIterator, PlanIteratorState>(sctx, loc, children)
+ {}
+
+ virtual ~IsStopWordLangSupportedIterator();
+
+ void accept(PlanIterVisitor& v) const;
+
+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
+};
+
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+/**
+ *
+ * Author:
+ */
+class IsThesaurusLangSupportedIterator : public NaryBaseIterator<IsThesaurusLangSupportedIterator, PlanIteratorState>
+{
+public:
+ SERIALIZABLE_CLASS(IsThesaurusLangSupportedIterator);
+
+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(IsThesaurusLangSupportedIterator,
+ NaryBaseIterator<IsThesaurusLangSupportedIterator, PlanIteratorState>);
+
+ void serialize( ::zorba::serialization::Archiver& ar)
+ {
+ serialize_baseclass(ar,
+ (NaryBaseIterator<IsThesaurusLangSupportedIterator, PlanIteratorState>*)this);
+ }
+
+ IsThesaurusLangSupportedIterator(
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& children)
+ :
+ NaryBaseIterator<IsThesaurusLangSupportedIterator, PlanIteratorState>(sctx, loc, children)
+ {}
+
+ virtual ~IsThesaurusLangSupportedIterator();
+
+ void accept(PlanIterVisitor& v) const;
+
+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
+};
+
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+/**
+ *
+ * Author:
+ */
+class IsTokenizerLangSupportedIterator : public NaryBaseIterator<IsTokenizerLangSupportedIterator, PlanIteratorState>
+{
+public:
+ SERIALIZABLE_CLASS(IsTokenizerLangSupportedIterator);
+
+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(IsTokenizerLangSupportedIterator,
+ NaryBaseIterator<IsTokenizerLangSupportedIterator, PlanIteratorState>);
+
+ void serialize( ::zorba::serialization::Archiver& ar)
+ {
+ serialize_baseclass(ar,
+ (NaryBaseIterator<IsTokenizerLangSupportedIterator, PlanIteratorState>*)this);
+ }
+
+ IsTokenizerLangSupportedIterator(
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& children)
+ :
+ NaryBaseIterator<IsTokenizerLangSupportedIterator, PlanIteratorState>(sctx, loc, children)
+ {}
+
+ virtual ~IsTokenizerLangSupportedIterator();
+
+ void accept(PlanIterVisitor& v) const;
+
+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
+};
+
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+/**
+ *
+ * Author:
+ */
+class StemIterator : public NaryBaseIterator<StemIterator, PlanIteratorState>
+{
+public:
+ SERIALIZABLE_CLASS(StemIterator);
+
+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(StemIterator,
+ NaryBaseIterator<StemIterator, PlanIteratorState>);
+
+ void serialize( ::zorba::serialization::Archiver& ar)
+ {
+ serialize_baseclass(ar,
+ (NaryBaseIterator<StemIterator, PlanIteratorState>*)this);
+ }
+
+ StemIterator(
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& children)
+ :
+ NaryBaseIterator<StemIterator, PlanIteratorState>(sctx, loc, children)
+ {}
+
+ virtual ~StemIterator();
+
+ void accept(PlanIterVisitor& v) const;
+
+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
+};
+
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+/**
+ *
+ * Author:
+ */
+class StripDiacriticsIterator : public NaryBaseIterator<StripDiacriticsIterator, PlanIteratorState>
+{
+public:
+ SERIALIZABLE_CLASS(StripDiacriticsIterator);
+
+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(StripDiacriticsIterator,
+ NaryBaseIterator<StripDiacriticsIterator, PlanIteratorState>);
+
+ void serialize( ::zorba::serialization::Archiver& ar)
+ {
+ serialize_baseclass(ar,
+ (NaryBaseIterator<StripDiacriticsIterator, PlanIteratorState>*)this);
+ }
+
+ StripDiacriticsIterator(
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& children)
+ :
+ NaryBaseIterator<StripDiacriticsIterator, PlanIteratorState>(sctx, loc, children)
+ {}
+
+ virtual ~StripDiacriticsIterator();
+
+ void accept(PlanIterVisitor& v) const;
+
+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
+};
+
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+/**
+ *
+ * Author:
+ */
+class ThesaurusLookupIteratorState : public PlanIteratorState
+{
+public:
+ zstring phrase_; //
+ zstring relationship_; //
+ internal::Thesaurus::level_type at_least_; //
+ internal::Thesaurus::level_type at_most_; //
+ internal::Thesaurus::ptr thesaurus_; //
+ internal::Thesaurus::iterator::ptr tresult_; //
+
+ ThesaurusLookupIteratorState();
+
+ ~ThesaurusLookupIteratorState();
+
+ void reset(PlanState&);
+};
+
+class ThesaurusLookupIterator : public NaryBaseIterator<ThesaurusLookupIterator, ThesaurusLookupIteratorState>
+{
+public:
+ SERIALIZABLE_CLASS(ThesaurusLookupIterator);
+
+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(ThesaurusLookupIterator,
+ NaryBaseIterator<ThesaurusLookupIterator, ThesaurusLookupIteratorState>);
+
+ void serialize( ::zorba::serialization::Archiver& ar)
+ {
+ serialize_baseclass(ar,
+ (NaryBaseIterator<ThesaurusLookupIterator, ThesaurusLookupIteratorState>*)this);
+ }
+
+ ThesaurusLookupIterator(
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& children)
+ :
+ NaryBaseIterator<ThesaurusLookupIterator, ThesaurusLookupIteratorState>(sctx, loc, children)
+ {}
+
+ virtual ~ThesaurusLookupIterator();
+
+ void accept(PlanIterVisitor& v) const;
+
+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
+
+ void resetImpl(PlanState&) const;
+};
+
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+/**
+ *
+ * Author:
+ */
+class TokenizeIteratorState : public PlanIteratorState
+{
+public:
+ store::Item_t doc_item_; //
+ FTTokenIterator_t doc_tokens_; //
+
+ TokenizeIteratorState();
+
+ ~TokenizeIteratorState();
+
+ void reset(PlanState&);
+};
+
+class TokenizeIterator : public NaryBaseIterator<TokenizeIterator, TokenizeIteratorState>
+{
+protected:
+ store::Item_t token_qname_; //
+ store::Item_t lang_qname_; //
+ store::Item_t para_qname_; //
+ store::Item_t sent_qname_; //
+ store::Item_t value_qname_; //
+ store::Item_t ref_qname_; //
+public:
+ SERIALIZABLE_CLASS(TokenizeIterator);
+
+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(TokenizeIterator,
+ NaryBaseIterator<TokenizeIterator, TokenizeIteratorState>);
+
+ void serialize( ::zorba::serialization::Archiver& ar);
+
+ TokenizeIterator(
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& children)
+ ;
+
+ virtual ~TokenizeIterator();
+
+public:
+ void initMembers();
+ void accept(PlanIterVisitor& v) const;
+
+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
+
+ void resetImpl(PlanState&) const;
+};
+
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+/**
+ *
+ * Author:
+ */
+class TokenizerPropertiesIterator : public NaryBaseIterator<TokenizerPropertiesIterator, PlanIteratorState>
+{
+public:
+ SERIALIZABLE_CLASS(TokenizerPropertiesIterator);
+
+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(TokenizerPropertiesIterator,
+ NaryBaseIterator<TokenizerPropertiesIterator, PlanIteratorState>);
+
+ void serialize( ::zorba::serialization::Archiver& ar)
+ {
+ serialize_baseclass(ar,
+ (NaryBaseIterator<TokenizerPropertiesIterator, PlanIteratorState>*)this);
+ }
+
+ TokenizerPropertiesIterator(
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& children)
+ :
+ NaryBaseIterator<TokenizerPropertiesIterator, PlanIteratorState>(sctx, loc, children)
+ {}
+
+ virtual ~TokenizerPropertiesIterator();
+
+ void accept(PlanIterVisitor& v) const;
+
+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
+};
+
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+/**
+ *
+ * Author:
+ */
+class TokenizeStringIteratorState : public PlanIteratorState
+{
+public:
+ FTTokenSeqIterator string_tokens_; //
+
+ TokenizeStringIteratorState();
+
+ ~TokenizeStringIteratorState();
+
+ void reset(PlanState&);
+};
+
+class TokenizeStringIterator : public NaryBaseIterator<TokenizeStringIterator, TokenizeStringIteratorState>
+{
+public:
+ SERIALIZABLE_CLASS(TokenizeStringIterator);
+
+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(TokenizeStringIterator,
+ NaryBaseIterator<TokenizeStringIterator, TokenizeStringIteratorState>);
+
+ void serialize( ::zorba::serialization::Archiver& ar)
+ {
+ serialize_baseclass(ar,
+ (NaryBaseIterator<TokenizeStringIterator, TokenizeStringIteratorState>*)this);
+ }
+
+ TokenizeStringIterator(
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& children)
+ :
+ NaryBaseIterator<TokenizeStringIterator, TokenizeStringIteratorState>(sctx, loc, children)
+ {}
+
+ virtual ~TokenizeStringIterator();
+
+ void accept(PlanIterVisitor& v) const;
+
+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
+
+ void resetImpl(PlanState&) const;
+};
+
+#endif
+
+}
+#endif
+/*
+ * Local variables:
+ * mode: c++
+ * End:
+ */
=== modified file 'src/runtime/full_text/stemmer.cpp'
--- src/runtime/full_text/stemmer.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/stemmer.cpp 2012-05-16 22:51:21 +0000
@@ -43,7 +43,8 @@
return default_provider;
}
-Stemmer::ptr StemmerProvider::get_stemmer( iso639_1::type lang ) const {
+bool StemmerProvider::getStemmer( iso639_1::type lang,
+ Stemmer::ptr *result ) const {
typedef unique_ptr<SnowballStemmer const> cache_ptr;
static cache_ptr cached_stemmers[ iso639_1::NUM_ENTRIES ];
@@ -56,7 +57,12 @@
cache_ptr &ptr_ref = cached_stemmers[ lang ];
if ( !ptr_ref )
ptr_ref.reset( SnowballStemmer::create( lang ) );
- return Stemmer::ptr( ptr_ref.get() );
+ if ( ptr_ref ) {
+ if ( result )
+ result->reset( ptr_ref.get() );
+ return true;
+ }
+ return false;
}
///////////////////////////////////////////////////////////////////////////////
=== modified file 'src/runtime/full_text/stemmer.h'
--- src/runtime/full_text/stemmer.h 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/stemmer.h 2012-05-16 22:51:21 +0000
@@ -41,11 +41,28 @@
ztd::destroy_delete<Stemmer const> > ptr;
/**
+ * Various properties of this %Stemmer.
+ */
+ struct Properties {
+ /**
+ * The URI that uniquely identifies this %Stemmer.
+ */
+ char const * uri;
+ };
+
+ /**
* Destroys this %Stemmer.
*/
virtual void destroy() const = 0;
/**
+ * Gets the Properties of this %Stemmer.
+ *
+ * @param result The Properties to populate.
+ */
+ virtual void properties( Properties *result ) const = 0;
+
+ /**
* Gets the stem of the given word.
*
* @param word The word to stem.
@@ -74,13 +91,15 @@
static StemmerProvider const& get_default();
/**
- * Gets an instance of a Stemmer for the given language.
+ * Gets a Stemmer for the given language.
*
- * @param lang The language for the stemmer.
- * @return Returns said Stemmer or \c nullptr if no stemmer is availabe for
- * the given language.
+ * @param lang The language to get a Stemmer for.
+ * @param s If not \c null, set to point to a Stemmer for \a lang.
+ * @return Returns \c true only if this provider can provide a stemmer for
+ * \a lang.
*/
- virtual Stemmer::ptr get_stemmer( locale::iso639_1::type lang ) const;
+ virtual bool getStemmer( locale::iso639_1::type lang,
+ Stemmer::ptr *s = 0 ) const;
};
///////////////////////////////////////////////////////////////////////////////
=== modified file 'src/runtime/full_text/stemmer/sb_stemmer.cpp'
--- src/runtime/full_text/stemmer/sb_stemmer.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/stemmer/sb_stemmer.cpp 2012-05-16 22:51:21 +0000
@@ -32,18 +32,21 @@
static bool is_lang_supported( iso639_1::type lang ) {
using namespace iso639_1;
switch ( lang ) {
- case da:
- case de:
- case en:
- case es:
- case fi:
- case hu:
- case it:
- case nl:
- case no:
- case pt:
- case sv:
- case ru:
+ case da: // Danish
+ case de: // German
+ case en: // English
+ case es: // Spanish
+ case fi: // Finnish
+ case fr: // French
+ case hu: // Hungarian
+ case it: // Italian
+ case nl: // Dutch
+ case no: // Norwegian
+ case pt: // Portuguese
+ case ro: // Romanian
+ case ru: // Russian
+ case sv: // Swedish
+ case tr: // Turkish
return true;
default:
return false;
@@ -70,7 +73,11 @@
// Do nothing since built-in stemmers are cached for re-use.
}
-void SnowballStemmer::stem( zstring const &word, iso639_1::type lang,
+void SnowballStemmer::properties( Properties *p ) const {
+ p->uri = "http://www.zorba-xquery.com/full-text/stemmer/snowball";;
+}
+
+void SnowballStemmer::stem( zstring const &word, iso639_1::type,
zstring *result ) const {
//
// We need a mutex since the libstemmer library is not thread-safe.
=== modified file 'src/runtime/full_text/stemmer/sb_stemmer.h'
--- src/runtime/full_text/stemmer/sb_stemmer.h 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/stemmer/sb_stemmer.h 2012-05-16 22:51:21 +0000
@@ -43,6 +43,7 @@
// inherited
void destroy() const;
+ void properties( Properties* ) const;
void stem( zstring const &word, locale::iso639_1::type lang,
zstring *result ) const;
=== modified file 'src/runtime/full_text/thesauri/wn_thesaurus.cpp'
--- src/runtime/full_text/thesauri/wn_thesaurus.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/thesauri/wn_thesaurus.cpp 2012-05-16 22:51:21 +0000
@@ -23,6 +23,8 @@
#include <zorba/util/path.h>
+#include <context/static_context.h>
+
#include "util/cxx_util.h"
#include "util/fs_util.h"
#include "util/less.h"
@@ -56,6 +58,18 @@
////////// Helper functions ///////////////////////////////////////////////////
/**
+ * Appends the name of the file Zorba uses for a WordNet thesaurus files.
+ *
+ * @param path The path to append to.
+ * @param lang The language of the thesaurus file.
+ */
+static void append_wordnet_file( zstring &path, iso639_1::type lang ) {
+ fs::append( path, "wordnet-" );
+ path += iso639_1::string_of[ lang ];
+ path += ".zth";
+}
+
+/**
* "Fixes" the "at most" parameter. The Full Text specification section 3.4.3
* says in part:
*
@@ -70,8 +84,10 @@
* broad, hence if at_most specifies "all levels" (max int), clamp it at 2
* (which seems to work well in practice).
*/
-inline ft_int fix_at_most( ft_int at_most ) {
- return at_most == numeric_limits<ft_int>::max() ? 2 : at_most;
+inline internal::Thesaurus::level_type
+fix_at_most( internal::Thesaurus::level_type at_most ) {
+ return at_most == numeric_limits<internal::Thesaurus::level_type>::max() ?
+ 2 : at_most;
}
/**
@@ -191,9 +207,7 @@
for ( bool loop = true; loop; ) {
switch ( fs::get_type( path ) ) {
case fs::directory:
- fs::append( path, "wordnet-" );
- path += iso639_1::string_of[ iso639_1::en ];
- path += ".zth";
+ append_wordnet_file( path, iso639_1::en );
break;
case fs::file:
loop = false;
@@ -216,7 +230,7 @@
*
* @param relationship The XQuery thesaurus relationship.
* @param lang The language of the relationship.
- * @return Returns the corresponding Wordnet pointer type.
+ * @return Returns the corresponding WordNet pointer type.
*/
static pointer::type map_xquery_rel( zstring const &relationship,
iso639_1::type lang ) {
@@ -233,8 +247,8 @@
thesaurus::iterator::LevelMarker = make_pair( ~0u, iso2788::neutral );
thesaurus::iterator::iterator( thesaurus const &t, char const *p,
- pointer::type ptr_type, ft_int at_least,
- ft_int at_most ) :
+ pointer::type ptr_type, level_type at_least,
+ level_type at_most ) :
thesaurus_( t ), query_ptr_type_( ptr_type ),
at_least_( at_least ), at_most_( fix_at_most( at_most ) ), level_( 0 )
{
@@ -506,7 +520,7 @@
thesaurus::iterator::ptr
thesaurus::lookup( zstring const &phrase, zstring const &relationship,
- ft_int at_least, ft_int at_most ) const {
+ level_type at_least, level_type at_most ) const {
iterator::ptr result;
# if DEBUG_FT_THESAURUS
cout << "==================================================" << endl;
@@ -524,6 +538,62 @@
///////////////////////////////////////////////////////////////////////////////
+provider::provider( zstring const &path ) : path_( path ) {
+ ZORBA_ASSERT( !path.empty() );
+}
+
+bool provider::getThesaurus( iso639_1::type lang,
+ internal::Thesaurus::ptr *t ) const {
+#ifdef ZORBA_WITH_FILE_ACCESS
+ zstring file_path;
+
+ switch ( lang ) {
+ case iso639_1::unknown:
+ lang = iso639_1::en;
+ // no break;
+ case iso639_1::en:
+ file_path = path_;
+ append_wordnet_file( file_path, lang );
+ break;
+ default:
+ return false;
+ }
+
+ //
+ // We want to look for the WordNet thesaurus file on the library path.
+ // Unfortunately every static_context can have its own library path and we
+ // don't have direct access to the query's static_context here. So, for now
+ // we only look on the root static_context's library path.
+ //
+ static_context &sctx = GENV.getRootStaticContext();
+ std::vector<zstring> lib_path_components;
+ sctx.get_full_lib_path( lib_path_components );
+ MUTATE_EACH( std::vector<zstring>, path, lib_path_components ) {
+ fs::append( *path, file_path );
+ if ( fs::get_type( *path ) == fs::file ) {
+ if ( t )
+ t->reset( new thesaurus( *path, lang ) );
+ return true;
+ }
+ }
+ return false;
+#else
+ switch ( lang ) {
+ case iso639_1::unknown:
+ lang = iso639_1::en;
+ // no break;
+ case iso639_1::en:
+ if ( t )
+ t->reset();
+ return true;
+ default:
+ return false;
+ }
+#endif /* ZORBA_WITH_FILE_ACCESS */
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
} // namespace wordnet
} // namespace zorba
/* vim:set et sw=2 ts=2: */
=== modified file 'src/runtime/full_text/thesauri/wn_thesaurus.h'
--- src/runtime/full_text/thesauri/wn_thesaurus.h 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/thesauri/wn_thesaurus.h 2012-05-16 22:51:21 +0000
@@ -34,7 +34,7 @@
///////////////////////////////////////////////////////////////////////////////
/**
- * A %wordnet::thesaurus is an ft_thesaurus for Wordnet.
+ * A %wordnet::thesaurus is a Thesaurus for Wordnet.
* See: http://wordnet.princeton.edu/
*/
class thesaurus : public internal::Thesaurus {
@@ -44,7 +44,8 @@
// inherited
void destroy() const;
- iterator::ptr lookup( zstring const&, zstring const&, ft_int, ft_int ) const;
+ iterator::ptr lookup( zstring const&, zstring const&, level_type,
+ level_type ) const;
private:
/**
@@ -86,7 +87,7 @@
private:
iterator( thesaurus const&, char const *lemma, pointer::type,
- ft_int at_least, ft_int at_most );
+ level_type at_least, level_type at_most );
~iterator();
thesaurus const &thesaurus_;
@@ -97,8 +98,8 @@
*/
pointer::type query_ptr_type_;
- ft_int const at_least_, at_most_;
- ft_int level_;
+ level_type const at_least_, at_most_;
+ level_type level_;
typedef std::pair<synset_id_t,iso2788::rel_dir> candidate_t;
typedef std::deque<candidate_t> candidate_queue_t;
@@ -124,6 +125,29 @@
///////////////////////////////////////////////////////////////////////////////
+/**
+ * A %wordnet::provider is a ThesaurusProvider for Wordnet.
+ */
+class provider : public internal::ThesaurusProvider {
+public:
+ /**
+ * Constructs a %provider.
+ *
+ * @param path The relative path of where the wordnet-LL.zth file is located
+ * (where LL is the ISO 639-1 language code of the language).
+ */
+ provider( zstring const &path );
+
+ // inherited
+ bool getThesaurus( locale::iso639_1::type,
+ internal::Thesaurus::ptr* = nullptr ) const;
+
+private:
+ zstring const path_;
+};
+
+///////////////////////////////////////////////////////////////////////////////
+
} // namespace wordnet
} // namespace zorba
=== modified file 'src/runtime/full_text/thesauri/xqftts_thesaurus.cpp'
--- src/runtime/full_text/thesauri/xqftts_thesaurus.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/thesauri/xqftts_thesaurus.cpp 2012-05-16 22:51:21 +0000
@@ -60,8 +60,8 @@
make_pair( static_cast<synonym*>(0), iso2788::neutral );
thesaurus::iterator::iterator( thesaurus_t const &t, zstring const &phrase,
- zstring const &rel_string, ft_int at_least,
- ft_int at_most ) :
+ zstring const &rel_string, level_type at_least,
+ level_type at_most ) :
thesaurus_( t ), at_least_( at_least ), at_most_( at_most ), level_( 1 )
{
using namespace iso2788;
@@ -217,7 +217,7 @@
thesaurus::iterator::ptr
thesaurus::lookup( zstring const &phrase, zstring const &relationship,
- ft_int at_least, ft_int at_most ) const {
+ level_type at_least, level_type at_most ) const {
# if DEBUG_THESAURUS
cout << "==================================================" << endl;
cout << "query phrase: " << phrase << endl;
@@ -364,6 +364,31 @@
///////////////////////////////////////////////////////////////////////////////
+provider::provider( zstring const &path ) : path_( path ) {
+}
+
+bool provider::getThesaurus( iso639_1::type lang,
+ internal::Thesaurus::ptr *t ) const {
+ switch ( lang ) {
+ case iso639_1::unknown:
+ lang = iso639_1::en;
+ // no break;
+ case iso639_1::en:
+ if ( t ) {
+#ifdef ZORBA_WITH_FILE_ACCESS
+ t->reset( new thesaurus( path_, lang ) );
+#else
+ t->reset();
+#endif /* ZORBA_WITH_FILE_ACCESS */
+ }
+ return true;
+ default:
+ return false;
+ }
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
} // namespace xqftts
} // namespace zorba
/* vim:set et sw=2 ts=2: */
=== modified file 'src/runtime/full_text/thesauri/xqftts_thesaurus.h'
--- src/runtime/full_text/thesauri/xqftts_thesaurus.h 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/thesauri/xqftts_thesaurus.h 2012-05-16 22:51:21 +0000
@@ -44,7 +44,8 @@
// inherited
void destroy() const;
- iterator::ptr lookup( zstring const&, zstring const&, ft_int, ft_int ) const;
+ iterator::ptr lookup( zstring const&, zstring const&, level_type,
+ level_type ) const;
private:
//
@@ -123,13 +124,14 @@
private:
iterator( thesaurus_t const&, zstring const &phrase,
- zstring const &relationship, ft_int at_least, ft_int at_most );
+ zstring const &relationship, level_type at_least,
+ level_type at_most );
~iterator();
thesaurus_t const &thesaurus_;
- ft_int const at_least_, at_most_;
- ft_int level_;
+ level_type const at_least_, at_most_;
+ level_type level_;
typedef std::pair<synonym const*,iso2788::rel_dir> candidate_t;
typedef std::deque<candidate_t> candidate_queue_t;
@@ -155,6 +157,28 @@
///////////////////////////////////////////////////////////////////////////////
+/**
+ * A %xqftts::provider is a ThesaurusProvider for XQFTTS.
+ */
+class provider : public internal::ThesaurusProvider {
+public:
+ /**
+ * Constructs a %provider.
+ *
+ * @param path The absolute path of the thesaurus XML file.
+ */
+ provider( zstring const &path );
+
+ // inherited
+ bool getThesaurus( locale::iso639_1::type,
+ internal::Thesaurus::ptr* = nullptr ) const;
+
+private:
+ zstring const path_;
+};
+
+///////////////////////////////////////////////////////////////////////////////
+
} // namespace xqftts
} // namespace zorba
=== modified file 'src/runtime/full_text/thesaurus.cpp'
--- src/runtime/full_text/thesaurus.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/thesaurus.cpp 2012-05-16 22:51:21 +0000
@@ -31,6 +31,7 @@
#include "thesaurus.h"
#ifdef ZORBA_WITH_FILE_ACCESS
# include "thesauri/wn_thesaurus.h"
+# include "zorbatypes/URI.h"
#endif
#include "thesauri/xqftts_thesaurus.h"
@@ -57,7 +58,8 @@
type const DEFAULT = wordnet;
/**
- * Given a thesaurus implementation name, finds its corresponding type.
+ * Given a thesaurus implementation name (as identified by the URI scheme),
+ * finds its corresponding type.
*
* @param name The thesaurus implementation's name.
* @return Returns the implementation's type or \c unknown.
@@ -66,7 +68,6 @@
typedef map<char const*,type> impl_map_t;
static impl_map_t impl_map;
if ( impl_map.empty() ) {
- impl_map[ "default" ] = DEFAULT;
impl_map[ "wordnet" ] = wordnet;
impl_map[ "xqftts" ] = xqftts;
}
@@ -78,28 +79,6 @@
///////////////////////////////////////////////////////////////////////////////
-/**
- * Parses a thesaurus mapping string. A mapping string is of the form:
- *
- * [implementation_name|]URI
- *
- * @param mapping The mapping to parse.
- * @param t A pointer to receive the implementation type.
- * @param uri A pointer to the string to receive the URI.
- */
-static void parse_mapping( zstring const &mapping, thesaurus_impl::type *t,
- zstring *uri ) {
- zstring impl_name;
- if ( zorba::ztd::split( mapping, '|', &impl_name, uri ) ) {
- *t = thesaurus_impl::find( impl_name );
- } else {
- *t = thesaurus_impl::DEFAULT;
- *uri = mapping;
- }
-}
-
-///////////////////////////////////////////////////////////////////////////////
-
Thesaurus::iterator::~iterator() {
// out-of-line since it's virtual
}
@@ -112,36 +91,41 @@
Resource*
ThesaurusURLResolver::resolveURL( zstring const &url, EntityData const *data ) {
+ // Only resolve thesaurus URLs
if ( data->getKind() != internal::EntityData::THESAURUS )
return nullptr;
- ThesaurusEntityData const *const t_data =
- dynamic_cast<ThesaurusEntityData const*>( data );
- iso639_1::type const lang = t_data->getLanguage();
-
- thesaurus_impl::type t_impl;
- zstring mapped_url;
- parse_mapping( url, &t_impl, &mapped_url );
-
- zstring t_path;
- switch ( uri::get_scheme( mapped_url ) ) {
- case uri::file:
- case uri::none:
- t_path = fs::get_normalized_path( mapped_url );
- break;
- default:
- throw XQUERY_EXCEPTION(
- zerr::ZXQP0004_NOT_IMPLEMENTED,
- ERROR_PARAMS( ZED( NonFileThesaurusURI ) )
- );
- }
-
- switch ( t_impl ) {
+
+ zstring const url_copy(
+ url == "##default" ? "wordnet://wordnet.princeton.edu/": url
+ );
+
+ zstring scheme_name;
+ if ( !uri::get_scheme( url_copy, &scheme_name ) )
+ return nullptr;
+
+ switch ( thesaurus_impl::find( scheme_name ) ) {
+ case thesaurus_impl::xqftts: {
+ //
+ // Currently, we presume that an "xqftts:" URL should be used exactly
+ // like a "file:" URL.
+ //
+ zstring t_uri( url_copy );
+ t_uri.replace( 0, 6, "file" ); // xqftts -> file
+ zstring const t_path( fs::get_normalized_path( t_uri ) );
+ return new xqftts::provider( t_path );
+ }
# ifdef ZORBA_WITH_FILE_ACCESS
- case thesaurus_impl::wordnet:
- return new wordnet::thesaurus( t_path, lang );
+ case thesaurus_impl::wordnet: {
+ //
+ // Wordnet, on the other hand, needs to find its data file in Zorba's
+ // library path using the mangled form of the original URI. So, mangle
+ // here for convenience.
+ //
+ URI const t_uri( url_copy );
+ zstring const t_path( t_uri.toPathNotation() );
+ return new wordnet::provider( t_path );
+ }
# endif /* ZORBA_WITH_FILE_ACCESS */
- case thesaurus_impl::xqftts:
- return new xqftts::thesaurus( t_path, lang );
default:
throw XQUERY_EXCEPTION( err::FTST0018, ERROR_PARAMS( url ) );
}
=== modified file 'src/runtime/full_text/thesaurus.h'
--- src/runtime/full_text/thesaurus.h 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/thesaurus.h 2012-05-16 22:51:21 +0000
@@ -34,9 +34,14 @@
/**
* A %Thesaurus is the abstract base class for thesaurus implementations.
*/
-class Thesaurus : public internal::Resource {
+class Thesaurus {
public:
- typedef std::unique_ptr<Thesaurus,ztd::destroy_delete<Thesaurus> > ptr;
+ typedef ft_int level_type;
+
+ typedef std::unique_ptr<
+ Thesaurus const,ztd::destroy_delete<Thesaurus const>
+ >
+ ptr;
/**
* An %iterator is used to iterate over lookup results.
@@ -82,8 +87,9 @@
* the phrase was not found.
*/
virtual iterator::ptr lookup( zstring const &phrase,
- zstring const &relationship, ft_int at_least,
- ft_int at_most ) const = 0;
+ zstring const &relationship,
+ level_type at_least,
+ level_type at_most ) const = 0;
protected:
Thesaurus() { }
@@ -97,6 +103,26 @@
///////////////////////////////////////////////////////////////////////////////
+/**
+ * A %ThesaurusProvider is-a Resource for providing thesauri for a given
+ * language.
+ */
+class ThesaurusProvider : public internal::Resource {
+public:
+ /**
+ * Gets a Thesaurus for the given language.
+ *
+ * @param lang The desired language of the thesaurus.
+ * @param t If not \c null, set to point to a Thesaurus for \a lang.
+ * @return Returns \c true only if this provider can provide a thesaurus for
+ * \a lang.
+ */
+ virtual bool getThesaurus( locale::iso639_1::type lang,
+ Thesaurus::ptr *t = nullptr ) const = 0;
+};
+
+///////////////////////////////////////////////////////////////////////////////
+
} // namespace internal
} // namespace zorba
#endif /* ZORBA_THESAURUS_H */
=== modified file 'src/runtime/full_text/tokenizer.cpp'
--- src/runtime/full_text/tokenizer.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/tokenizer.cpp 2012-05-16 22:51:21 +0000
@@ -15,24 +15,98 @@
*/
#include "stdafx.h"
+#include <zorba/item.h>
+#include <zorba/iterator.h>
+#include <zorba/store_consts.h>
#include <zorba/tokenizer.h>
+#include <zorba/zorba_string.h>
+
+#include "diagnostics/assert.h"
+#include "store/api/store.h"
+#include "system/globalenv.h"
+#include "zorbamisc/ns_consts.h"
+#include "zorbautils/locale.h"
+
+using namespace zorba::locale;
namespace zorba {
///////////////////////////////////////////////////////////////////////////////
-Tokenizer::Tokenizer( Numbers &no, int trace_options ) :
- trace_options_( trace_options ),
- no_( &no )
-{
-}
-
Tokenizer::~Tokenizer() {
// out-of-line since it's virtual
}
-void Tokenizer::element( Item const&, int ) {
- // do nothing
+bool Tokenizer::find_lang_attribute( Item const &item, iso639_1::type *lang ) {
+ bool found_lang = false;
+ if ( item.getNodeKind() == store::StoreConsts::elementNode ) {
+ Iterator_t i( item.getAttributes() );
+ i->open();
+ for ( Item attr; i->next( attr ); ) {
+ Item qname;
+ if ( attr.getNodeName( qname ) &&
+ qname.getLocalName() == "lang" && qname.getNamespace() == XML_NS ) {
+ *lang = locale::find_lang( attr.getStringValue().c_str() );
+ found_lang = true;
+ break;
+ }
+ }
+ i->close();
+ }
+ return found_lang;
+}
+
+void Tokenizer::item( Item const &item, bool entering ) {
+ if ( entering && item.isNode() &&
+ item.getNodeKind() == store::StoreConsts::elementNode ) {
+ ++numbers().para;
+ }
+}
+
+void Tokenizer::tokenize_node_impl( Item const &item, iso639_1::type lang,
+ Callback &callback, bool tokenize_acp ) {
+ if ( item.isNode() ) {
+ Iterator_t i;
+ Tokenizer *t_raw = this;
+ Tokenizer::ptr t_ptr;
+
+ this->item( item, true );
+ callback.item( item, true );
+
+ switch ( item.getNodeKind() ) {
+ case store::StoreConsts::elementNode:
+ if ( find_lang_attribute( item, &lang ) ) {
+ TokenizerProvider const *const p = GENV_STORE.getTokenizerProvider();
+ ZORBA_ASSERT( p );
+ if ( !p->getTokenizer( lang, numbers_, &t_ptr ) )
+ break;
+ t_raw = t_ptr.get();
+ }
+ // no break;
+
+ case store::StoreConsts::documentNode:
+ i = item.getChildren();
+ i->open();
+ for ( Item child; i->next( child ); )
+ t_raw->tokenize_node_impl( child, lang, callback, false );
+ i->close();
+ break;
+
+ case store::StoreConsts::attributeNode:
+ case store::StoreConsts::commentNode:
+ case store::StoreConsts::piNode:
+ if ( !tokenize_acp )
+ break;
+ case store::StoreConsts::textNode: {
+ String const s( item.getStringValue() );
+ tokenize_string( s.data(), s.size(), lang, false, callback, &item );
+ break;
+ }
+ } // switch
+
+ this->item( item, false );
+ callback.item( item, false );
+ }
}
Tokenizer::Numbers::Numbers() {
@@ -44,6 +118,10 @@
// out-of-line since it's virtual
}
+void Tokenizer::Callback::item( Item const&, bool ) {
+ // out-of-line since it's virtual
+}
+
///////////////////////////////////////////////////////////////////////////////
TokenizerProvider::~TokenizerProvider() {
=== modified file 'src/runtime/indexing/doc_indexer.cpp'
--- src/runtime/indexing/doc_indexer.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/indexing/doc_indexer.cpp 2012-05-16 22:51:21 +0000
@@ -84,7 +84,7 @@
QueryLoc::null,
tmp);
- ulong numEntries = delta.size();
+ csize numEntries = delta.size();
store::Item_t domainNode;
store::IndexKey* key = NULL;
@@ -96,13 +96,11 @@
//std::cout << domainNode.getp() << " " << key << std::endl;
- for (ulong i = 0; i < theNumColumns; ++i)
+ for (csize i = 0; i < theNumColumns; ++i)
{
if (!thePlanWrapper->next((*key)[i]))
- throw ZORBA_EXCEPTION(
- zerr::ZXQP0003_INTERNAL_ERROR,
- ERROR_PARAMS( ZED( IncompleteKeyInIndexRefresh ) )
- );
+ throw ZORBA_EXCEPTION(zerr::ZXQP0003_INTERNAL_ERROR,
+ ERROR_PARAMS(ZED(IncompleteKeyInIndexRefresh)));
}
delta.resize(numEntries + 1);
=== modified file 'src/runtime/json/common.h'
--- src/runtime/json/common.h 2012-02-08 15:45:53 +0000
+++ src/runtime/json/common.h 2012-05-16 22:51:21 +0000
@@ -56,21 +56,26 @@
typedef std::ostream& (*std_omanip_type)(std::ostream&);
-inline std::ostream& if_do( std::ostream &o, bool expr, std_omanip_type fn ) {
+inline std::ostream& if_do_impl( std::ostream &o, bool expr,
+ std_omanip_type fn ) {
if ( expr )
o << fn;
return o;
}
-DEF_OMANIP2( if_do, bool, std_omanip_type )
+DEF_OMANIP2( if_do_impl, bool, std_omanip_type )
+// A macro with a !! is used to suppress a warning from MSVC++.
+#define if_do(EXPR,FN) if_do_impl( !!(EXPR), (FN) )
#define if_indent(WS,FN) if_do( (WS) == whitespace::indent, FN )
-inline std::ostream& if_emit( std::ostream &o, bool expr, char c ) {
+inline std::ostream& if_emit_impl( std::ostream &o, bool expr, char c ) {
if ( expr )
o << c;
return o;
}
-DEF_OMANIP2( if_emit, bool, char )
+DEF_OMANIP2( if_emit_impl, bool, char )
+// A macro with a !! is used to suppress a warning from MSVC++.
+#define if_emit(EXPR,C) if_emit_impl( !!(EXPR), (C) )
///////////////////////////////////////////////////////////////////////////////
=== modified file 'src/runtime/nodes/nodes_impl.cpp'
--- src/runtime/nodes/nodes_impl.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/nodes/nodes_impl.cpp 2012-05-16 22:51:21 +0000
@@ -628,5 +628,122 @@
STACK_END (state);
}
+/*******************************************************************************
+********************************************************************************/
+int getNodePosition(store::Item_t aNode)
+{
+ int count = 1;
+ store::Iterator_t lIterator = aNode->getParent()->getChildren();
+ store::Item_t lItem;
+ lIterator->open();
+ while(lIterator->next(lItem))
+ {
+ if(lItem->getNodeKind() == aNode->getNodeKind())
+ if(lItem->equals(aNode))
+ break;
+ else
+ count++;
+ }
+ lIterator->close();
+ return count;
+}
+
+bool FnPathIterator::nextImpl(store::Item_t& result, PlanState& planState) const
+{
+ store::Item_t inNode;
+ store::Item_t nodeName;
+ store::NsBindings nsBindings;
+ zstring path;
+ zstring temp;
+ zstring zNamespace;
+ zstring zLocalName;
+ zstring zPosition;
+ bool rootIsDocument = false;
+
+ PlanIteratorState* state;
+ DEFAULT_STACK_INIT(PlanIteratorState, state, planState);
+
+ if (consumeNext(inNode, theChildren[0], planState));
+ {
+ do
+ {
+ switch (inNode->getNodeKind())
+ {
+ case store::StoreConsts::documentNode:
+ temp = path;
+ path = "/";
+ path += temp;
+ rootIsDocument = true;
+ break;
+ case store::StoreConsts::elementNode:
+ nodeName = inNode->getNodeName();
+ zNamespace = nodeName->getNamespace();
+ zLocalName = nodeName->getLocalName();
+ zPosition = ztd::to_string(getNodePosition(inNode));
+ temp = path;
+ path = "\""+zNamespace+"\":"+zLocalName+"["+zPosition.c_str()+"]";
+ path += temp;
+ break;
+ case store::StoreConsts::attributeNode:
+ nodeName = inNode->getNodeName();
+ zNamespace =nodeName->getNamespace();
+ zLocalName = nodeName->getLocalName();
+ if(zNamespace != "")
+ {
+ temp = path;
+ path = "@\""+zNamespace+"\":"+zLocalName;
+ path += temp;
+ }
+ else
+ {
+ temp = path;
+ path = "@"+zLocalName;
+ path += temp;
+ }
+ break;
+ case store::StoreConsts::textNode:
+ zPosition = ztd::to_string(getNodePosition(inNode));
+ temp = path;
+ path = "text()["+zPosition+"]";
+ path += temp;
+ break;
+ case store::StoreConsts::commentNode:
+ zPosition = ztd::to_string(getNodePosition(inNode));
+ temp = path;
+ path = "comment()["+zPosition+"]";
+ path += temp;
+ break;
+ default:
+ if(inNode->isProcessingInstruction())
+ {
+ nodeName = inNode->getNodeName();
+ zLocalName = nodeName->getLocalName();
+ zPosition = ztd::to_string(getNodePosition(inNode));
+ temp = path;
+ path = "processing-instruction("+zLocalName+")["+zPosition+"]";
+ path += temp;
+ }
+ break;
+ }
+ inNode = inNode->getParent();
+
+ if(inNode && inNode->getNodeKind() != store::StoreConsts::documentNode)
+ {
+ temp = path;
+ path = "/";
+ path += temp;
+ }
+
+ } while (inNode);
+
+ if(rootIsDocument)
+ STACK_PUSH(GENV_ITEMFACTORY->createString(result, path), state);
+ else
+ throw XQUERY_EXCEPTION(err::FODC0001, ERROR_PARAMS("fn:path"), ERROR_LOC(loc));
+ }
+
+ STACK_END (state);
+}
+
} // namespace zorba
/* vim:set et sw=2 ts=2: */
=== modified file 'src/runtime/nodes/pregenerated/nodes.cpp'
--- src/runtime/nodes/pregenerated/nodes.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/nodes/pregenerated/nodes.cpp 2012-05-16 22:51:21 +0000
@@ -474,6 +474,28 @@
// </LeastCommonAncestor>
+// <FnPathIterator>
+FnPathIterator::class_factory<FnPathIterator>
+FnPathIterator::g_class_factory;
+
+
+void FnPathIterator::accept(PlanIterVisitor& v) const {
+ v.beginVisit(*this);
+
+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
+ for ( ; lIter != lEnd; ++lIter ){
+ (*lIter)->accept(v);
+ }
+
+ v.endVisit(*this);
+}
+
+FnPathIterator::~FnPathIterator() {}
+
+// </FnPathIterator>
+
+
}
=== modified file 'src/runtime/nodes/pregenerated/nodes.h'
--- src/runtime/nodes/pregenerated/nodes.h 2012-04-24 12:39:38 +0000
+++ src/runtime/nodes/pregenerated/nodes.h 2012-05-16 22:51:21 +0000
@@ -720,6 +720,40 @@
};
+/**
+ *
+ * Author: Zorba Team
+ */
+class FnPathIterator : public NaryBaseIterator<FnPathIterator, PlanIteratorState>
+{
+public:
+ SERIALIZABLE_CLASS(FnPathIterator);
+
+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(FnPathIterator,
+ NaryBaseIterator<FnPathIterator, PlanIteratorState>);
+
+ void serialize( ::zorba::serialization::Archiver& ar)
+ {
+ serialize_baseclass(ar,
+ (NaryBaseIterator<FnPathIterator, PlanIteratorState>*)this);
+ }
+
+ FnPathIterator(
+ static_context* sctx,
+ const QueryLoc& loc,
+ std::vector<PlanIter_t>& children)
+ :
+ NaryBaseIterator<FnPathIterator, PlanIteratorState>(sctx, loc, children)
+ {}
+
+ virtual ~FnPathIterator();
+
+ void accept(PlanIterVisitor& v) const;
+
+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
+};
+
+
}
#endif
/*
=== modified file 'src/runtime/numerics/numerics_impl.cpp'
--- src/runtime/numerics/numerics_impl.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/numerics/numerics_impl.cpp 2012-05-16 22:51:21 +0000
@@ -427,6 +427,8 @@
zstring infinity;
zstring NaN;
zstring minus;
+
+ zstring pictureString; // The original picture string, used for debugging and error reporting
class PartInfo
{
@@ -500,14 +502,14 @@
// returns an error if there are two or more instances of the given pattern in the string
-static void errorIfTwoOrMore(zstring const& part, const char* sep, QueryLoc& loc)
+static void errorIfTwoOrMore(zstring const& part, const char* sep, FormatNumberInfo& info)
{
zstring::size_type const pos = part.find(sep);
if (pos != zstring::npos)
{
if (part.find(sep, strlen(sep), pos+1) != zstring::npos)
- throw XQUERY_EXCEPTION(err::XTDE1310, ERROR_LOC(loc));
+ throw XQUERY_EXCEPTION(err::FODF1310, ERROR_PARAMS(info.pictureString, ZED(FormatNumberDuplicates), sep), ERROR_LOC(info.loc));
}
}
@@ -521,19 +523,19 @@
if (str.empty())
return;
- errorIfTwoOrMore(str, info.percent.c_str(), info.loc);
- errorIfTwoOrMore(str, info.per_mille.c_str(), info.loc);
+ errorIfTwoOrMore(str, info.percent.c_str(), info);
+ errorIfTwoOrMore(str, info.per_mille.c_str(), info);
if (str.find(info.percent.c_str()) != zstring::npos &&
str.find(info.per_mille.c_str()) != zstring::npos)
{
- throw XQUERY_EXCEPTION(err::XTDE1310, ERROR_LOC(info.loc));
+ throw XQUERY_EXCEPTION(err::FODF1310, ERROR_PARAMS(info.pictureString, ZED(FormatNumberPercentPermille)), ERROR_LOC(info.loc));
}
if (str.find(info.digit_sign.c_str()) == zstring::npos &&
str.find(info.zero_digit.c_str()) == zstring::npos)
{
- throw XQUERY_EXCEPTION(err::XTDE1310, ERROR_LOC(info.loc));
+ throw XQUERY_EXCEPTION(err::FODF1310, ERROR_PARAMS(info.pictureString, ZED(FormatNumberAtLeastOneOptionalOrDecimal)), ERROR_LOC(info.loc));
}
// get grouping separators
@@ -563,15 +565,16 @@
start += delta;
}
- if (first_digit_sign != -1 && last_zero_sign != -1
- &&
- ((!fractional && first_digit_sign > last_zero_sign)
- ||
- (fractional && first_digit_sign < last_zero_sign)))
- throw XQUERY_EXCEPTION(err::XTDE1310, ERROR_LOC(info.loc));
+ if (first_digit_sign != -1 && last_zero_sign != -1)
+ {
+ if (!fractional && first_digit_sign > last_zero_sign)
+ throw XQUERY_EXCEPTION(err::FODF1310, ERROR_PARAMS(info.pictureString, ZED(FormatNumberIntegerPart)), ERROR_LOC(info.loc));
+ else if (fractional && first_digit_sign < last_zero_sign)
+ throw XQUERY_EXCEPTION(err::FODF1310, ERROR_PARAMS(info.pictureString, ZED(FormatNumberFractionalPart)), ERROR_LOC(info.loc));
+ }
if (part.grouping_pos.size() > 0 && part.grouping_pos[0] == 0)
- throw XQUERY_EXCEPTION(err::XTDE1310, ERROR_LOC(info.loc));
+ throw XQUERY_EXCEPTION(err::FODF1310, ERROR_PARAMS(info.pictureString, ZED(FormatNumberGroupingAdjacentToDecimal)), ERROR_LOC(info.loc));
if (part.grouping_pos.size() > 0)
{
@@ -606,7 +609,7 @@
if (str.empty())
return;
- errorIfTwoOrMore(str, info.decimal_separator.c_str(), info.loc);
+ errorIfTwoOrMore(str, info.decimal_separator.c_str(), info);
zstring::size_type pos = str.find(info.decimal_separator.c_str());
if (pos != zstring::npos)
{
@@ -648,18 +651,18 @@
}
-static void parsePicture(zstring& picture, FormatNumberInfo& info)
+static void parsePicture(FormatNumberInfo& info)
{
- errorIfTwoOrMore(picture, info.pattern_separator.c_str(), info.loc);
+ errorIfTwoOrMore(info.pictureString, info.pattern_separator.c_str(), info);
- zstring::size_type pos = picture.find(info.pattern_separator.c_str());
+ zstring::size_type pos = info.pictureString.find(info.pattern_separator.c_str());
if (pos != zstring::npos)
{
- info.pos_subpicture.str = picture.substr(0, pos);
- info.neg_subpicture.str = picture.substr(pos+1, picture.size() - pos);
+ info.pos_subpicture.str = info.pictureString.substr(0, pos);
+ info.neg_subpicture.str = info.pictureString.substr(pos+1, info.pictureString.size() - pos);
}
else
- info.pos_subpicture.str = picture;
+ info.pos_subpicture.str = info.pictureString;
parseSubpicture(info.pos_subpicture, info);
if (info.neg_subpicture.str.empty())
@@ -862,7 +865,6 @@
FormatNumberIterator::nextImpl(store::Item_t& result, PlanState& planState) const
{
zstring resultString;
- zstring pictureString;
store::Item_t numberItem, pictureItem, formatName;
FormatNumberInfo info;
DecimalFormat_t df_t;
@@ -945,8 +947,8 @@
info.readFormat(df_t);
- pictureString = pictureItem->getStringValue();
- parsePicture(pictureString, info);
+ info.pictureString = pictureItem->getStringValue();
+ parsePicture(info);
formatNumber(resultString, result, info, theSctx->get_typemanager(), loc);
STACK_PUSH (GENV_ITEMFACTORY->createString(result, resultString), state);
=== modified file 'src/runtime/parsing_and_serializing/parse_fragment_impl.cpp'
--- src/runtime/parsing_and_serializing/parse_fragment_impl.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/parsing_and_serializing/parse_fragment_impl.cpp 2012-05-16 22:51:21 +0000
@@ -170,11 +170,7 @@
}
}
-/*******************************************************************************
-
-********************************************************************************/
-
-void FnParseXmlFragmentIteratorState::reset(PlanState& planState)
+void FnZorbaParseXmlFragmentIteratorState::reset(PlanState& planState)
{
PlanIteratorState::reset(planState);
theFragmentStream.reset();
@@ -184,20 +180,15 @@
docUri = "";
}
-
-/*******************************************************************************
-
-********************************************************************************/
-
-bool FnParseXmlFragmentIterator::nextImpl(store::Item_t& result, PlanState& planState) const
+bool FnZorbaParseXmlFragmentIterator::nextImpl(store::Item_t& result, PlanState& planState) const
{
store::Store& lStore = GENV.getStore();
zstring docString;
store::Item_t tempItem;
bool validated = true;
- FnParseXmlFragmentIteratorState* state;
- DEFAULT_STACK_INIT(FnParseXmlFragmentIteratorState, state, planState);
+ FnZorbaParseXmlFragmentIteratorState* state;
+ DEFAULT_STACK_INIT(FnZorbaParseXmlFragmentIteratorState, state, planState);
if (consumeNext(result, theChildren[0].getp(), planState))
{
@@ -311,5 +302,66 @@
}
+/*******************************************************************************
+ 14.9.2 fn:parse-xml-fragment
+********************************************************************************/
+/*
+bool FnParseXmlFragmentIterator::nextImpl(store::Item_t& result, PlanState& planState) const
+{
+ zstring docString;
+
+ FnParseXmlFragmentIteratorState* state;
+ DEFAULT_STACK_INIT(FnParseXmlFragmentIteratorState, state, planState);
+
+ if (consumeNext(result, theChildren[0].getp(), planState))
+ {
+ if (result->isStreamable())
+ {
+ state->theFragmentStream.theStream = &result->getStream();
+ }
+ else
+ {
+ result->getStringValue2(docString);
+ state->theFragmentStream.theIss = new std::istringstream(docString.c_str());
+ state->theFragmentStream.theStream = state->theFragmentStream.theIss;
+ }
+
+ state->theProperties.setBaseUri(theSctx->get_base_uri());
+ state->baseUri = state->theProperties.getBaseUri();
+
+ state->theProperties.setParseExternalParsedEntity(true);
+
+ while ( ! state->theFragmentStream.stream_is_consumed() )
+ {
+ try {
+ state->theProperties.setStoreDocument(false);
+ result = GENV.getStore().loadDocument(state->baseUri, state->docUri, state->theFragmentStream, state->theProperties);
+ } catch (ZorbaException const& e) {
+ if( ! state->theProperties.getNoError())
+ throw XQUERY_EXCEPTION(err::FODC0006, ERROR_PARAMS("fn:parse-xml-fragment()", e.what() ), ERROR_LOC(loc));
+ else
+ result = NULL;
+ }
+
+ if (result == NULL)
+ continue;
+
+ STACK_PUSH(true, state);
+ } // while
+ } // if
+
+ STACK_END(state)
+}
+
+void FnParseXmlFragmentIteratorState::reset(PlanState& planState)
+{
+ PlanIteratorState::reset(planState);
+ theFragmentStream.reset();
+ theProperties.reset();
+ theProperties.setStoreDocument(false);
+ baseUri = "";
+ docUri = "";
+}
+*/
} /* namespace zorba */
=== modified file 'src/runtime/parsing_and_serializing/parsing_and_serializing_impl.cpp'
--- src/runtime/parsing_and_serializing/parsing_and_serializing_impl.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/parsing_and_serializing/parsing_and_serializing_impl.cpp 2012-05-16 22:51:21 +0000
@@ -118,7 +118,6 @@
STACK_END (state);
}
-
/*******************************************************************************
14.9.2 fn:serialize
********************************************************************************/
=== modified file 'src/runtime/parsing_and_serializing/pregenerated/parse_fragment.cpp'
--- src/runtime/parsing_and_serializing/pregenerated/parse_fragment.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/parsing_and_serializing/pregenerated/parse_fragment.cpp 2012-05-16 22:51:21 +0000
@@ -32,12 +32,12 @@
namespace zorba {
-// <FnParseXmlFragmentIterator>
-FnParseXmlFragmentIterator::class_factory<FnParseXmlFragmentIterator>
-FnParseXmlFragmentIterator::g_class_factory;
-
-
-void FnParseXmlFragmentIterator::accept(PlanIterVisitor& v) const {
+// <FnZorbaParseXmlFragmentIterator>
+FnZorbaParseXmlFragmentIterator::class_factory<FnZorbaParseXmlFragmentIterator>
+FnZorbaParseXmlFragmentIterator::g_class_factory;
+
+
+void FnZorbaParseXmlFragmentIterator::accept(PlanIterVisitor& v) const {
v.beginVisit(*this);
std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
@@ -49,17 +49,17 @@
v.endVisit(*this);
}
-FnParseXmlFragmentIterator::~FnParseXmlFragmentIterator() {}
-
-FnParseXmlFragmentIteratorState::FnParseXmlFragmentIteratorState() {}
-
-FnParseXmlFragmentIteratorState::~FnParseXmlFragmentIteratorState() {}
-
-
-void FnParseXmlFragmentIteratorState::init(PlanState& planState) {
+FnZorbaParseXmlFragmentIterator::~FnZorbaParseXmlFragmentIterator() {}
+
+FnZorbaParseXmlFragmentIteratorState::FnZorbaParseXmlFragmentIteratorState() {}
+
+FnZorbaParseXmlFragmentIteratorState::~FnZorbaParseXmlFragmentIteratorState() {}
+
+
+void FnZorbaParseXmlFragmentIteratorState::init(PlanState& planState) {
PlanIteratorState::init(planState);
}
-// </FnParseXmlFragmentIterator>
+// </FnZorbaParseXmlFragmentIterator>
=== modified file 'src/runtime/parsing_and_serializing/pregenerated/parse_fragment.h'
--- src/runtime/parsing_and_serializing/pregenerated/parse_fragment.h 2012-04-24 12:39:38 +0000
+++ src/runtime/parsing_and_serializing/pregenerated/parse_fragment.h 2012-05-16 22:51:21 +0000
@@ -39,7 +39,7 @@
* fn-zorba-xml:parse
* Author: Zorba Team
*/
-class FnParseXmlFragmentIteratorState : public PlanIteratorState
+class FnZorbaParseXmlFragmentIteratorState : public PlanIteratorState
{
public:
FragmentIStream theFragmentStream; //the input fragment
@@ -47,37 +47,37 @@
zstring baseUri; //
zstring docUri; //
- FnParseXmlFragmentIteratorState();
+ FnZorbaParseXmlFragmentIteratorState();
- ~FnParseXmlFragmentIteratorState();
+ ~FnZorbaParseXmlFragmentIteratorState();
void init(PlanState&);
void reset(PlanState&);
};
-class FnParseXmlFragmentIterator : public NaryBaseIterator<FnParseXmlFragmentIterator, FnParseXmlFragmentIteratorState>
+class FnZorbaParseXmlFragmentIterator : public NaryBaseIterator<FnZorbaParseXmlFragmentIterator, FnZorbaParseXmlFragmentIteratorState>
{
public:
- SERIALIZABLE_CLASS(FnParseXmlFragmentIterator);
+ SERIALIZABLE_CLASS(FnZorbaParseXmlFragmentIterator);
- SERIALIZABLE_CLASS_CONSTRUCTOR2T(FnParseXmlFragmentIterator,
- NaryBaseIterator<FnParseXmlFragmentIterator, FnParseXmlFragmentIteratorState>);
+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(FnZorbaParseXmlFragmentIterator,
+ NaryBaseIterator<FnZorbaParseXmlFragmentIterator, FnZorbaParseXmlFragmentIteratorState>);
void serialize( ::zorba::serialization::Archiver& ar)
{
serialize_baseclass(ar,
- (NaryBaseIterator<FnParseXmlFragmentIterator, FnParseXmlFragmentIteratorState>*)this);
+ (NaryBaseIterator<FnZorbaParseXmlFragmentIterator, FnZorbaParseXmlFragmentIteratorState>*)this);
}
- FnParseXmlFragmentIterator(
+ FnZorbaParseXmlFragmentIterator(
static_context* sctx,
const QueryLoc& loc,
std::vector<PlanIter_t>& children)
:
- NaryBaseIterator<FnParseXmlFragmentIterator, FnParseXmlFragmentIteratorState>(sctx, loc, children)
+ NaryBaseIterator<FnZorbaParseXmlFragmentIterator, FnZorbaParseXmlFragmentIteratorState>(sctx, loc, children)
{}
- virtual ~FnParseXmlFragmentIterator();
+ virtual ~FnZorbaParseXmlFragmentIterator();
void accept(PlanIterVisitor& v) const;
=== modified file 'src/runtime/spec/accessors/accessors.xml'
--- src/runtime/spec/accessors/accessors.xml 2012-04-24 12:39:38 +0000
+++ src/runtime/spec/accessors/accessors.xml 2012-05-16 22:51:21 +0000
@@ -62,13 +62,16 @@
<zorba:function>
+ <zorba:signature localname="nilled" prefix="fn" version="3.0">
+ <zorba:output>xs:boolean</zorba:output>
+ </zorba:signature>
<zorba:signature localname="nilled" prefix="fn">
<zorba:param>node()?</zorba:param>
<zorba:output>xs:boolean?</zorba:output>
</zorba:signature>
-
+
<zorba:methods>
- <zorba:mustCopyInputNodes value="false"/>
+ <zorba:mustCopyInputNodes value="true"/>
</zorba:methods>
</zorba:function>
=== modified file 'src/runtime/spec/codegen-cpp.xq'
--- src/runtime/spec/codegen-cpp.xq 2012-04-24 12:39:38 +0000
+++ src/runtime/spec/codegen-cpp.xq 2012-05-16 22:51:21 +0000
@@ -95,7 +95,8 @@
if (exists($iter/@preprocessorGuard))
then
- concat($gen:newline, "#endif")
+ concat($gen:newline, "#endif
+")
else
""
)
@@ -194,7 +195,7 @@
if (count($function/zorba:signature) = 0)
then
(: TODO user fn:error :)
- 'Error: could not find "prefix" and "localname" attributes for "zorba:function" element'
+ 'Error: could not find \"prefix\" and \"localname\" attributes for \"zorba:function\" element'
else
let $name := concat(local:function-name($function), $suffix)
let $ret := if($iter/@name = "") then "return NULL;"
@@ -275,7 +276,8 @@
($gen:newline,
if (exists($iter/@preprocessorGuard))
then
- concat($gen:newline, $iter/@preprocessorGuard)
+ concat($gen:newline, $iter/@preprocessorGuard, "
+")
else
"",
$gen:indent,
@@ -336,7 +338,13 @@
'),', $gen:newline, gen:indent(4),
'FunctionConsts::', gen:function-kind($sig) ,');',
$gen:newline, $gen:newline, $gen:indent,
- '}', $gen:newline, $gen:newline
+ '}', $gen:newline, $gen:newline,
+ if (exists($iter/@preprocessorGuard))
+ then
+ concat($gen:newline, "#endif
+")
+ else
+ ""
),
''),
'')
@@ -351,7 +359,7 @@
then
$tmp/@uri
else (: TODO user fn:error :)
- 'Error: could not find "prefix" and "localname" attributes for "zorba:function" element'
+ 'Error: could not find \"prefix\" and \"localname\" attributes for \"zorba:function\" element'
};
=== modified file 'src/runtime/spec/codegen-h.xq'
--- src/runtime/spec/codegen-h.xq 2012-04-24 12:39:38 +0000
+++ src/runtime/spec/codegen-h.xq 2012-05-16 22:51:21 +0000
@@ -146,7 +146,7 @@
if(count($function/zorba:signature) = 0)
then
(: TODO user fn:error :)
- 'Error: could not find "prefix" and "localname" attributes for "zorba:function" element'
+ 'Error: could not find \"prefix\" and \"localname\" attributes for \"zorba:function\" element'
else
local:create-function-XQuery-30($iter, $function)
(: local:create-function-arity($iter, $function, xs:integer(1)) :)
=== modified file 'src/runtime/spec/collections/collections.xml'
--- src/runtime/spec/collections/collections.xml 2012-04-24 12:39:38 +0000
+++ src/runtime/spec/collections/collections.xml 2012-05-16 22:51:21 +0000
@@ -1324,32 +1324,32 @@
14.8.5 fn:uri-collection
********************************************************************************/
-->
- <zorba:iterator name="FnURICollectionIterator">
- <zorba:description author="Zorba Team">
- Returns a sequence of xs:anyURI values representing the document URIs of the
- documents in a collection.
- </zorba:description>
-
- <zorba:function>
-
- <zorba:signature localname="uri-collection" prefix="fn" version="3.0">
- <zorba:output>xs:anyURI*</zorba:output>
- </zorba:signature>
-
- <zorba:signature localname="uri-collection" prefix="fn" version="3.0">
- <zorba:param>xs:string?</zorba:param>
- <zorba:output>xs:anyURI*</zorba:output>
- </zorba:signature>
-
- </zorba:function>
-
- <zorba:state generateInit="false" generateReset="false" generateDestructor="false">
- <zorba:member type="store::Iterator_t" name="theIterator"
- brief="the current iterator"/>
- <zorba:member type="bool" name="theIteratorOpened" defaultValue="false"
- brief="flag indicating whether theIterator was opened"/>
- </zorba:state>
+<zorba:iterator name="FnURICollectionIterator">
+ <zorba:description author="Zorba Team">
+ Returns a sequence of xs:anyURI values representing the document URIs of the
+ documents in a collection.
+ </zorba:description>
+
+ <zorba:function>
+
+ <zorba:signature localname="uri-collection" prefix="fn" version="3.0">
+ <zorba:output>xs:anyURI*</zorba:output>
+ </zorba:signature>
+
+ <zorba:signature localname="uri-collection" prefix="fn" version="3.0">
+ <zorba:param>xs:string?</zorba:param>
+ <zorba:output>xs:anyURI*</zorba:output>
+ </zorba:signature>
- </zorba:iterator>
+ </zorba:function>
+
+ <zorba:state generateInit="false" generateReset="false" generateDestructor="false">
+ <zorba:member type="store::Iterator_t" name="theIterator"
+ brief="the current iterator"/>
+ <zorba:member type="bool" name="theIteratorOpened" defaultValue="false"
+ brief="flag indicating whether theIterator was opened"/>
+ </zorba:state>
+
+</zorba:iterator>
</zorba:iterators>
=== added directory 'src/runtime/spec/full_text'
=== added file 'src/runtime/spec/full_text/ft_module.xml'
--- src/runtime/spec/full_text/ft_module.xml 1970-01-01 00:00:00 +0000
+++ src/runtime/spec/full_text/ft_module.xml 2012-05-16 22:51:21 +0000
@@ -0,0 +1,218 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<zorba:iterators
+ xmlns:zorba="http://www.zorba-xquery.com";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://www.zorba-xquery.com ../runtime.xsd">
+
+<zorba:header>
+ <zorba:include form="Quoted">runtime/full_text/ft_token_seq_iterator.h</zorba:include>
+ <zorba:include form="Quoted">runtime/full_text/thesaurus.h</zorba:include>
+</zorba:header>
+
+<zorba:source>
+ <zorba:include form="Quoted">store/api/iterator.h</zorba:include>
+</zorba:source>
+
+<zorba:iterator name="CurrentLangIterator"
+ preprocessorGuard="#ifndef ZORBA_NO_FULL_TEXT">
+ <zorba:function>
+ <zorba:signature localname="current-lang" prefix="full-text">
+ <zorba:output>xs:language</zorba:output>
+ </zorba:signature>
+ </zorba:function>
+</zorba:iterator>
+
+<zorba:iterator name="HostLangIterator"
+ preprocessorGuard="#ifndef ZORBA_NO_FULL_TEXT">
+ <zorba:function>
+ <zorba:signature localname="host-lang" prefix="full-text">
+ <zorba:output>xs:language</zorba:output>
+ </zorba:signature>
+ </zorba:function>
+</zorba:iterator>
+
+<zorba:iterator name="IsStemLangSupportedIterator"
+ preprocessorGuard="#ifndef ZORBA_NO_FULL_TEXT">
+ <zorba:function>
+ <zorba:signature localname="is-stem-lang-supported" prefix="full-text">
+ <zorba:param>xs:language</zorba:param>
+ <zorba:output>xs:boolean</zorba:output>
+ </zorba:signature>
+ </zorba:function>
+</zorba:iterator>
+
+<zorba:iterator name="IsStopWordIterator"
+ preprocessorGuard="#ifndef ZORBA_NO_FULL_TEXT">
+ <zorba:function>
+ <zorba:signature localname="is-stop-word" prefix="full-text">
+ <zorba:param>xs:string</zorba:param> <!-- word -->
+ <zorba:output>xs:boolean</zorba:output>
+ </zorba:signature>
+ <zorba:signature localname="is-stop-word" prefix="full-text">
+ <zorba:param>xs:string</zorba:param> <!-- word -->
+ <zorba:param>xs:language</zorba:param> <!-- lang -->
+ <zorba:output>xs:boolean</zorba:output>
+ </zorba:signature>
+ </zorba:function>
+</zorba:iterator>
+
+<zorba:iterator name="IsStopWordLangSupportedIterator"
+ preprocessorGuard="#ifndef ZORBA_NO_FULL_TEXT">
+ <zorba:function>
+ <zorba:signature localname="is-stop-word-lang-supported" prefix="full-text">
+ <zorba:param>xs:language</zorba:param>
+ <zorba:output>xs:boolean</zorba:output>
+ </zorba:signature>
+ </zorba:function>
+</zorba:iterator>
+
+<zorba:iterator name="IsThesaurusLangSupportedIterator"
+ preprocessorGuard="#ifndef ZORBA_NO_FULL_TEXT">
+ <zorba:function>
+ <zorba:signature localname="is-thesaurus-lang-supported" prefix="full-text">
+ <zorba:param>xs:language</zorba:param>
+ <zorba:output>xs:boolean</zorba:output>
+ </zorba:signature>
+ <zorba:signature localname="is-thesaurus-lang-supported" prefix="full-text">
+ <zorba:param>xs:string</zorba:param> <!-- URI -->
+ <zorba:param>xs:language</zorba:param>
+ <zorba:output>xs:boolean</zorba:output>
+ </zorba:signature>
+ </zorba:function>
+</zorba:iterator>
+
+<zorba:iterator name="IsTokenizerLangSupportedIterator"
+ preprocessorGuard="#ifndef ZORBA_NO_FULL_TEXT">
+ <zorba:function>
+ <zorba:signature localname="is-tokenizer-lang-supported" prefix="full-text">
+ <zorba:param>xs:language</zorba:param>
+ <zorba:output>xs:boolean</zorba:output>
+ </zorba:signature>
+ </zorba:function>
+</zorba:iterator>
+
+<zorba:iterator name="StemIterator"
+ preprocessorGuard="#ifndef ZORBA_NO_FULL_TEXT">
+ <zorba:function>
+ <zorba:signature localname="stem" prefix="full-text">
+ <zorba:param>xs:string</zorba:param> <!-- word -->
+ <zorba:output>xs:string</zorba:output>
+ </zorba:signature>
+ <zorba:signature localname="stem" prefix="full-text">
+ <zorba:param>xs:string</zorba:param> <!-- word -->
+ <zorba:param>xs:language</zorba:param> <!-- lang -->
+ <zorba:output>xs:string</zorba:output>
+ </zorba:signature>
+ </zorba:function>
+</zorba:iterator>
+
+<zorba:iterator name="StripDiacriticsIterator"
+ preprocessorGuard="#ifndef ZORBA_NO_FULL_TEXT">
+ <zorba:function>
+ <zorba:signature localname="strip-diacritics" prefix="full-text">
+ <zorba:param>xs:string</zorba:param> <!-- phrase -->
+ <zorba:output>xs:string</zorba:output>
+ </zorba:signature>
+ </zorba:function>
+</zorba:iterator>
+
+<zorba:iterator name="ThesaurusLookupIterator"
+ generateResetImpl="true"
+ preprocessorGuard="#ifndef ZORBA_NO_FULL_TEXT">
+ <zorba:function>
+ <zorba:signature localname="thesaurus-lookup" prefix="full-text">
+ <zorba:param>xs:string</zorba:param> <!-- phrase -->
+ <zorba:output>xs:string+</zorba:output>
+ </zorba:signature>
+ <zorba:signature localname="thesaurus-lookup" prefix="full-text">
+ <zorba:param>xs:string</zorba:param> <!-- URI -->
+ <zorba:param>xs:string</zorba:param> <!-- phrase -->
+ <zorba:output>xs:string+</zorba:output>
+ </zorba:signature>
+ <zorba:signature localname="thesaurus-lookup" prefix="full-text">
+ <zorba:param>xs:string</zorba:param> <!-- URI -->
+ <zorba:param>xs:string</zorba:param> <!-- phrase -->
+ <zorba:param>xs:language</zorba:param> <!-- lang -->
+ <zorba:output>xs:string+</zorba:output>
+ </zorba:signature>
+ <zorba:signature localname="thesaurus-lookup" prefix="full-text">
+ <zorba:param>xs:string</zorba:param> <!-- URI -->
+ <zorba:param>xs:string</zorba:param> <!-- phrase -->
+ <zorba:param>xs:language</zorba:param> <!-- lang -->
+ <zorba:param>xs:string</zorba:param> <!-- relationship -->
+ <zorba:output>xs:string+</zorba:output>
+ </zorba:signature>
+ <zorba:signature localname="thesaurus-lookup" prefix="full-text">
+ <zorba:param>xs:string</zorba:param> <!-- URI -->
+ <zorba:param>xs:string</zorba:param> <!-- phrase -->
+ <zorba:param>xs:language</zorba:param> <!-- lang -->
+ <zorba:param>xs:string</zorba:param> <!-- relationship -->
+ <zorba:param>xs:integer</zorba:param> <!-- level-least -->
+ <zorba:param>xs:integer</zorba:param> <!-- level-most -->
+ <zorba:output>xs:string+</zorba:output>
+ </zorba:signature>
+ </zorba:function>
+ <zorba:state generateInit="use-default">
+ <zorba:member type="zstring" name="phrase_"/>
+ <zorba:member type="zstring" name="relationship_"/>
+ <zorba:member type="internal::Thesaurus::level_type" name="at_least_"/>
+ <zorba:member type="internal::Thesaurus::level_type" name="at_most_"/>
+ <zorba:member type="internal::Thesaurus::ptr" name="thesaurus_"/>
+ <zorba:member type="internal::Thesaurus::iterator::ptr" name="tresult_"/>
+ </zorba:state>
+</zorba:iterator>
+
+<zorba:iterator name="TokenizeIterator"
+ generateResetImpl="true"
+ generateSerialize="false"
+ generateConstructor="false"
+ preprocessorGuard="#ifndef ZORBA_NO_FULL_TEXT">
+
+ <zorba:state generateInit="use-default">
+ <zorba:member type="store::Item_t" name="doc_item_"/>
+ <zorba:member type="FTTokenIterator_t" name="doc_tokens_"/>
+ </zorba:state>
+
+ <zorba:member type="store::Item_t" name="token_qname_"/>
+ <zorba:member type="store::Item_t" name="lang_qname_"/>
+ <zorba:member type="store::Item_t" name="para_qname_"/>
+ <zorba:member type="store::Item_t" name="sent_qname_"/>
+ <zorba:member type="store::Item_t" name="value_qname_"/>
+ <zorba:member type="store::Item_t" name="ref_qname_"/>
+
+ <zorba:method name="initMembers" return="void"/>
+
+</zorba:iterator>
+
+<zorba:iterator name="TokenizerPropertiesIterator"
+ preprocessorGuard="#ifndef ZORBA_NO_FULL_TEXT">
+</zorba:iterator>
+
+<zorba:iterator name="TokenizeStringIterator"
+ generateResetImpl="true"
+ preprocessorGuard="#ifndef ZORBA_NO_FULL_TEXT">
+
+ <zorba:function>
+
+ <zorba:signature localname="tokenize-string" prefix="full-text">
+ <zorba:param>xs:string</zorba:param> <!-- string -->
+ <zorba:output>xs:string*</zorba:output>
+ </zorba:signature>
+
+ <zorba:signature localname="tokenize-string" prefix="full-text">
+ <zorba:param>xs:string</zorba:param> <!-- string -->
+ <zorba:param>xs:language</zorba:param> <!-- lang -->
+ <zorba:output>xs:string*</zorba:output>
+ </zorba:signature>
+
+ </zorba:function>
+
+ <zorba:state generateInit="use-default">
+ <zorba:member type="FTTokenSeqIterator" name="string_tokens_"/>
+ </zorba:state>
+
+</zorba:iterator>
+
+</zorba:iterators>
+<!-- vim:set et sw=2 ts=2: -->
=== modified file 'src/runtime/spec/iterator_h.xq'
--- src/runtime/spec/iterator_h.xq 2012-04-24 12:39:38 +0000
+++ src/runtime/spec/iterator_h.xq 2012-05-16 22:51:21 +0000
@@ -153,16 +153,21 @@
local:children-decl($iter),
local:add-constructor-param($iter),
')',
- $gen:newline, gen:indent(2), ': ',
- $gen:newline, gen:indent(2), $base, '(sctx, loc', local:children-args($iter),
- if ($iter/@base)
- then concat(', ',
- string-join(
- for $base-param in $iter/zorba:constructor/zorba:parameter[@base = "true"]
- return $base-param/@name, ', '))
- else "",
- local:add-constructor-param-2($iter),
- $gen:newline, gen:indent(1), '{}',
+ $gen:newline, gen:indent(2),
+ if (not(exists($iter/@generateConstructor)) or $iter/@generateConstructor = "true")
+ then concat(
+ ': ',
+ $gen:newline, gen:indent(2), $base, '(sctx, loc', local:children-args($iter),
+ if ($iter/@base)
+ then concat(', ',
+ string-join(
+ for $base-param in $iter/zorba:constructor/zorba:parameter[@base = "true"]
+ return $base-param/@name, ', '))
+ else "",
+ local:add-constructor-param-2($iter),
+ $gen:newline, gen:indent(1), '{}')
+ else
+ ';',
$gen:newline, $gen:newline
)
};
=== modified file 'src/runtime/spec/mappings.xml'
--- src/runtime/spec/mappings.xml 2012-04-24 12:39:38 +0000
+++ src/runtime/spec/mappings.xml 2012-05-16 22:51:21 +0000
@@ -82,6 +82,11 @@
define="ZORBA_STORE_DYNAMIC_UNORDERED_MAP_FN_NS"
prefix="zorba-store-data-structure-unordered-map"/>
+ <zorba:namespace
+ uri="http://www.zorba-xquery.com/modules/full-text";
+ define="ZORBA_FULL_TEXT_FN_NS"
+ prefix="full-text"/>
+
<zorba:namespace uri="http://www.zorba-xquery.com/modules/xqdoc";
define="ZORBA_XQDOC_FN_NS"
prefix="fn-zorba-xqdoc"/>
@@ -150,9 +155,9 @@
<zorba:type zorbaType="ANY_NODE">node()</zorba:type>
<zorba:type zorbaType="ELEMENT">element()</zorba:type>
-
<zorba:type zorbaType="ANY_ATOMIC">xs:anyAtomicType</zorba:type>
<zorba:type zorbaType="UNTYPED_ATOMIC">xs:untypedAtomic</zorba:type>
+
<zorba:type zorbaType="STRING">xs:string</zorba:type>
<zorba:type zorbaType="NORMALIZED_STRING">xs:normalizedString</zorba:type>
<zorba:type zorbaType="TOKEN">xs:token</zorba:type>
@@ -160,21 +165,25 @@
<zorba:type zorbaType="NMTOKEN">xs:NMTOKEN</zorba:type>
<zorba:type zorbaType="NAME">xs:Name</zorba:type>
<zorba:type zorbaType="NCNAME">xs:NCName</zorba:type>
+
<zorba:type zorbaType="ID">xs:ID</zorba:type>
<zorba:type zorbaType="IDREF">xs:IDREF</zorba:type>
+
<zorba:type zorbaType="ENTITY">xs:ENTITY</zorba:type>
+
<zorba:type zorbaType="DATETIME">xs:dateTime</zorba:type>
<zorba:type zorbaType="DATE">xs:date</zorba:type>
<zorba:type zorbaType="TIME">xs:time</zorba:type>
<zorba:type zorbaType="DURATION">xs:duration</zorba:type>
<zorba:type zorbaType="DT_DURATION">xs:dayTimeDuration</zorba:type>
<zorba:type zorbaType="YM_DURATION">xs:yearMonthDuration</zorba:type>
+
<zorba:type zorbaType="FLOAT">xs:float</zorba:type>
<zorba:type zorbaType="DOUBLE">xs:double</zorba:type>
<zorba:type zorbaType="DECIMAL">xs:decimal</zorba:type>
<zorba:type zorbaType="INTEGER">xs:integer</zorba:type>
<zorba:type zorbaType="NON_POSITIVE_INTEGER">xs:nonPositiveInteger</zorba:type>
- <zorba:type zorbaType="NEGATIVE_INTEGER">xs:nonNegativeInteger</zorba:type>
+ <zorba:type zorbaType="NEGATIVE_INTEGER">xs:negativeInteger</zorba:type>
<zorba:type zorbaType="LONG">xs:long</zorba:type>
<zorba:type zorbaType="INT">xs:int</zorba:type>
<zorba:type zorbaType="SHORT">xs:short</zorba:type>
@@ -185,14 +194,17 @@
<zorba:type zorbaType="UNSIGNED_SHORT">xs:unsignedShort</zorba:type>
<zorba:type zorbaType="UNSIGNED_BYTE">xs:unsignedByte</zorba:type>
<zorba:type zorbaType="POSITIVE_INTEGER">xs:positiveInteger</zorba:type>
+
<zorba:type zorbaType="GYEAR_MONTH">xs:gYearMonth</zorba:type>
<zorba:type zorbaType="GYEAR">xs:gYear</zorba:type>
<zorba:type zorbaType="GMONTH_DAY">xs:gMonthDay</zorba:type>
<zorba:type zorbaType="GDAY">xs:gDay</zorba:type>
<zorba:type zorbaType="GMONTH">xs:gMonth</zorba:type>
+
<zorba:type zorbaType="BOOLEAN">xs:boolean</zorba:type>
<zorba:type zorbaType="BASE64BINARY">xs:base64Binary</zorba:type>
<zorba:type zorbaType="HEXBINARY">xs:hexBinary</zorba:type>
+
<zorba:type zorbaType="ANY_URI">xs:anyURI</zorba:type>
<zorba:type zorbaType="QNAME">xs:QName</zorba:type>
<zorba:type zorbaType="NOTATION">xs:NOTATION</zorba:type>
=== modified file 'src/runtime/spec/nodes/nodes.xml'
--- src/runtime/spec/nodes/nodes.xml 2012-04-24 12:39:38 +0000
+++ src/runtime/spec/nodes/nodes.xml 2012-05-16 22:51:21 +0000
@@ -186,6 +186,11 @@
<zorba:description author="Zorba Team">fn:has-children</zorba:description>
<zorba:function>
+
+ <zorba:signature localname="has-children" prefix="fn" version="3.0">
+ <zorba:output>xs:boolean</zorba:output>
+ </zorba:signature>
+
<zorba:signature localname="has-children" prefix="fn" version="3.0">
<zorba:param>node()?</zorba:param>
<zorba:output>xs:boolean</zorba:output>
@@ -515,4 +520,26 @@
</zorba:iterator>
+<!--
+/*******************************************************************************
+********************************************************************************/
+-->
+<zorba:iterator name="FnPathIterator">
+
+ <zorba:description author="Zorba Team"></zorba:description>
+
+ <zorba:function>
+ <zorba:signature localname="path" prefix="fn" version="3.0">
+ <zorba:output>xs:string?</zorba:output>
+ </zorba:signature>
+
+ <zorba:signature localname="path" prefix="fn" version="3.0">
+ <zorba:param>node()?</zorba:param>
+ <zorba:output>xs:string?</zorba:output>
+ </zorba:signature>
+
+ </zorba:function>
+
+</zorba:iterator>
+
</zorba:iterators>
=== modified file 'src/runtime/spec/parsing_and_serializing/parse_fragment.xml'
--- src/runtime/spec/parsing_and_serializing/parse_fragment.xml 2012-04-24 12:39:38 +0000
+++ src/runtime/spec/parsing_and_serializing/parse_fragment.xml 2012-05-16 22:51:21 +0000
@@ -22,7 +22,7 @@
* 14.9.1 parse-xml:parse
********************************************************************************/
-->
- <zorba:iterator name="FnParseXmlFragmentIterator">
+ <zorba:iterator name="FnZorbaParseXmlFragmentIterator">
<zorba:description author="Zorba Team">fn-zorba-xml:parse</zorba:description>
@@ -46,6 +46,35 @@
</zorba:state>
</zorba:iterator>
-
-
+<!--
+/*******************************************************************************
+ * 14.9.2 fn:parse-xml-fragment
+********************************************************************************/
+-->
+
+<!--
+<zorba:iterator name="FnParseXmlFragmentIterator">
+
+ <zorba:description author="Zorba Team">fn:parse-xml-fragment</zorba:description>
+
+ <zorba:function>
+ <zorba:signature localname="parse-xml-fragment" prefix="fn">
+ <zorba:param>xs:string?</zorba:param>
+ <zorba:output>node()*</zorba:output>
+ </zorba:signature>
+
+ <zorba:methods>
+ <zorba:accessesDynCtx returnValue="true"/>
+ </zorba:methods>
+ </zorba:function>
+
+ <zorba:state generateReset="false">
+ <zorba:member type="FragmentIStream" name="theFragmentStream" brief="the input fragment"/>
+ <zorba:member type="store::LoadProperties" name="theProperties" brief="loader properties"/>
+ <zorba:member type="zstring" name="baseUri"/>
+ <zorba:member type="zstring" name="docUri"/>
+ </zorba:state>
+
+</zorba:iterator>
+-->
</zorba:iterators>
=== modified file 'src/runtime/spec/parsing_and_serializing/parsing_and_serializing.xml'
--- src/runtime/spec/parsing_and_serializing/parsing_and_serializing.xml 2012-04-24 12:39:38 +0000
+++ src/runtime/spec/parsing_and_serializing/parsing_and_serializing.xml 2012-05-16 22:51:21 +0000
@@ -43,10 +43,9 @@
</zorba:iterator>
-
<!--
/*******************************************************************************
- * 14.9.2 fn:serialize
+ * 14.9.3 fn:serialize
********************************************************************************/
-->
<zorba:iterator name="FnSerializeIterator">
@@ -79,5 +78,6 @@
</zorba:method>
</zorba:iterator>
-
+
+
</zorba:iterators>
=== modified file 'src/runtime/strings/strings_impl.cpp'
--- src/runtime/strings/strings_impl.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/strings/strings_impl.cpp 2012-05-16 22:51:21 +0000
@@ -1840,7 +1840,7 @@
store::NsBindings ns_binding;
zstring baseURI;
GENV_ITEMFACTORY->createQName(untyped_type_name,
- "http://www.w3.org/2001/XMLSchema";, "xs", "untyped");
+ XML_SCHEMA_NS, XML_SCHEMA_PREFIX, "untyped");
GENV_ITEMFACTORY->createQName(non_match_element_name,
static_context::W3C_FN_NS, "fn", "non-match");
GENV_ITEMFACTORY->createElementNode(non_match_elem, parent, non_match_element_name, untyped_type_name, false, false, ns_binding, baseURI);
@@ -1969,7 +1969,7 @@
store::NsBindings ns_binding;
zstring baseURI;
GENV_ITEMFACTORY->createQName(untyped_type_name,
- "http://www.w3.org/2001/XMLSchema";, "xs", "untyped");
+ XML_SCHEMA_NS, XML_SCHEMA_PREFIX, "untyped");
GENV_ITEMFACTORY->createQName(match_element_name,
static_context::W3C_FN_NS, "fn", "match");
store::Item_t match_elem;
@@ -2129,7 +2129,7 @@
store::NsBindings ns_binding;
zstring baseURI;
GENV_ITEMFACTORY->createQName(untyped_type_name,
- "http://www.w3.org/2001/XMLSchema";, "xs", "untyped");
+ XML_SCHEMA_NS, XML_SCHEMA_PREFIX, "untyped");
GENV_ITEMFACTORY->createQName(result_element_name,
static_context::W3C_FN_NS, "fn", "analyze-string-result");
GENV_ITEMFACTORY->createElementNode(result, NULL, result_element_name, untyped_type_name, false, false, ns_binding, baseURI);
=== modified file 'src/runtime/visitors/pregenerated/planiter_visitor.h'
--- src/runtime/visitors/pregenerated/planiter_visitor.h 2012-04-26 20:54:34 +0000
+++ src/runtime/visitors/pregenerated/planiter_visitor.h 2012-05-16 22:51:21 +0000
@@ -193,6 +193,45 @@
class FnPutIterator;
+#ifndef ZORBA_NO_FULL_TEXT
+ class CurrentLangIterator;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ class HostLangIterator;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ class IsStemLangSupportedIterator;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ class IsStopWordIterator;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ class IsStopWordLangSupportedIterator;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ class IsThesaurusLangSupportedIterator;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ class IsTokenizerLangSupportedIterator;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ class StemIterator;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ class StripDiacriticsIterator;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ class ThesaurusLookupIterator;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ class TokenizeIterator;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ class TokenizerPropertiesIterator;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ class TokenizeStringIterator;
+#endif
class FunctionNameIterator;
class FunctionArityIterator;
@@ -397,6 +436,8 @@
class LeastCommonAncestor;
+ class FnPathIterator;
+
class AbsIterator;
class CeilingIterator;
@@ -411,7 +452,7 @@
class FormatIntegerIterator;
- class FnParseXmlFragmentIterator;
+ class FnZorbaParseXmlFragmentIterator;
class FnParseXmlIterator;
@@ -862,6 +903,58 @@
virtual void beginVisit ( const FnPutIterator& ) = 0;
virtual void endVisit ( const FnPutIterator& ) = 0;
+#ifndef ZORBA_NO_FULL_TEXT
+ virtual void beginVisit ( const CurrentLangIterator& ) = 0;
+ virtual void endVisit ( const CurrentLangIterator& ) = 0;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ virtual void beginVisit ( const HostLangIterator& ) = 0;
+ virtual void endVisit ( const HostLangIterator& ) = 0;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ virtual void beginVisit ( const IsStemLangSupportedIterator& ) = 0;
+ virtual void endVisit ( const IsStemLangSupportedIterator& ) = 0;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ virtual void beginVisit ( const IsStopWordIterator& ) = 0;
+ virtual void endVisit ( const IsStopWordIterator& ) = 0;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ virtual void beginVisit ( const IsStopWordLangSupportedIterator& ) = 0;
+ virtual void endVisit ( const IsStopWordLangSupportedIterator& ) = 0;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ virtual void beginVisit ( const IsThesaurusLangSupportedIterator& ) = 0;
+ virtual void endVisit ( const IsThesaurusLangSupportedIterator& ) = 0;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ virtual void beginVisit ( const IsTokenizerLangSupportedIterator& ) = 0;
+ virtual void endVisit ( const IsTokenizerLangSupportedIterator& ) = 0;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ virtual void beginVisit ( const StemIterator& ) = 0;
+ virtual void endVisit ( const StemIterator& ) = 0;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ virtual void beginVisit ( const StripDiacriticsIterator& ) = 0;
+ virtual void endVisit ( const StripDiacriticsIterator& ) = 0;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ virtual void beginVisit ( const ThesaurusLookupIterator& ) = 0;
+ virtual void endVisit ( const ThesaurusLookupIterator& ) = 0;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ virtual void beginVisit ( const TokenizeIterator& ) = 0;
+ virtual void endVisit ( const TokenizeIterator& ) = 0;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ virtual void beginVisit ( const TokenizerPropertiesIterator& ) = 0;
+ virtual void endVisit ( const TokenizerPropertiesIterator& ) = 0;
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+ virtual void beginVisit ( const TokenizeStringIterator& ) = 0;
+ virtual void endVisit ( const TokenizeStringIterator& ) = 0;
+#endif
virtual void beginVisit ( const FunctionNameIterator& ) = 0;
virtual void endVisit ( const FunctionNameIterator& ) = 0;
@@ -1168,6 +1261,9 @@
virtual void beginVisit ( const LeastCommonAncestor& ) = 0;
virtual void endVisit ( const LeastCommonAncestor& ) = 0;
+ virtual void beginVisit ( const FnPathIterator& ) = 0;
+ virtual void endVisit ( const FnPathIterator& ) = 0;
+
virtual void beginVisit ( const AbsIterator& ) = 0;
virtual void endVisit ( const AbsIterator& ) = 0;
@@ -1189,8 +1285,8 @@
virtual void beginVisit ( const FormatIntegerIterator& ) = 0;
virtual void endVisit ( const FormatIntegerIterator& ) = 0;
- virtual void beginVisit ( const FnParseXmlFragmentIterator& ) = 0;
- virtual void endVisit ( const FnParseXmlFragmentIterator& ) = 0;
+ virtual void beginVisit ( const FnZorbaParseXmlFragmentIterator& ) = 0;
+ virtual void endVisit ( const FnZorbaParseXmlFragmentIterator& ) = 0;
virtual void beginVisit ( const FnParseXmlIterator& ) = 0;
virtual void endVisit ( const FnParseXmlIterator& ) = 0;
=== modified file 'src/runtime/visitors/pregenerated/printer_visitor.cpp'
--- src/runtime/visitors/pregenerated/printer_visitor.cpp 2012-04-24 12:39:38 +0000
+++ src/runtime/visitors/pregenerated/printer_visitor.cpp 2012-05-16 22:51:21 +0000
@@ -47,6 +47,7 @@
#include "runtime/errors_and_diagnostics/other_diagnostics.h"
#include "runtime/fetch/fetch.h"
#include "runtime/fnput/fnput.h"
+#include "runtime/full_text/ft_module.h"
#include "runtime/function_item/function_item_iter.h"
#include "runtime/indexing/ic_ddl.h"
#include "runtime/introspection/sctx.h"
@@ -1245,6 +1246,201 @@
}
// </FnPutIterator>
+#ifndef ZORBA_NO_FULL_TEXT
+// <CurrentLangIterator>
+void PrinterVisitor::beginVisit ( const CurrentLangIterator& a) {
+ thePrinter.startBeginVisit("CurrentLangIterator", ++theId);
+ printCommons( &a, theId );
+ thePrinter.endBeginVisit( theId );
+}
+
+void PrinterVisitor::endVisit ( const CurrentLangIterator& ) {
+ thePrinter.startEndVisit();
+ thePrinter.endEndVisit();
+}
+// </CurrentLangIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <HostLangIterator>
+void PrinterVisitor::beginVisit ( const HostLangIterator& a) {
+ thePrinter.startBeginVisit("HostLangIterator", ++theId);
+ printCommons( &a, theId );
+ thePrinter.endBeginVisit( theId );
+}
+
+void PrinterVisitor::endVisit ( const HostLangIterator& ) {
+ thePrinter.startEndVisit();
+ thePrinter.endEndVisit();
+}
+// </HostLangIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <IsStemLangSupportedIterator>
+void PrinterVisitor::beginVisit ( const IsStemLangSupportedIterator& a) {
+ thePrinter.startBeginVisit("IsStemLangSupportedIterator", ++theId);
+ printCommons( &a, theId );
+ thePrinter.endBeginVisit( theId );
+}
+
+void PrinterVisitor::endVisit ( const IsStemLangSupportedIterator& ) {
+ thePrinter.startEndVisit();
+ thePrinter.endEndVisit();
+}
+// </IsStemLangSupportedIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <IsStopWordIterator>
+void PrinterVisitor::beginVisit ( const IsStopWordIterator& a) {
+ thePrinter.startBeginVisit("IsStopWordIterator", ++theId);
+ printCommons( &a, theId );
+ thePrinter.endBeginVisit( theId );
+}
+
+void PrinterVisitor::endVisit ( const IsStopWordIterator& ) {
+ thePrinter.startEndVisit();
+ thePrinter.endEndVisit();
+}
+// </IsStopWordIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <IsStopWordLangSupportedIterator>
+void PrinterVisitor::beginVisit ( const IsStopWordLangSupportedIterator& a) {
+ thePrinter.startBeginVisit("IsStopWordLangSupportedIterator", ++theId);
+ printCommons( &a, theId );
+ thePrinter.endBeginVisit( theId );
+}
+
+void PrinterVisitor::endVisit ( const IsStopWordLangSupportedIterator& ) {
+ thePrinter.startEndVisit();
+ thePrinter.endEndVisit();
+}
+// </IsStopWordLangSupportedIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <IsThesaurusLangSupportedIterator>
+void PrinterVisitor::beginVisit ( const IsThesaurusLangSupportedIterator& a) {
+ thePrinter.startBeginVisit("IsThesaurusLangSupportedIterator", ++theId);
+ printCommons( &a, theId );
+ thePrinter.endBeginVisit( theId );
+}
+
+void PrinterVisitor::endVisit ( const IsThesaurusLangSupportedIterator& ) {
+ thePrinter.startEndVisit();
+ thePrinter.endEndVisit();
+}
+// </IsThesaurusLangSupportedIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <IsTokenizerLangSupportedIterator>
+void PrinterVisitor::beginVisit ( const IsTokenizerLangSupportedIterator& a) {
+ thePrinter.startBeginVisit("IsTokenizerLangSupportedIterator", ++theId);
+ printCommons( &a, theId );
+ thePrinter.endBeginVisit( theId );
+}
+
+void PrinterVisitor::endVisit ( const IsTokenizerLangSupportedIterator& ) {
+ thePrinter.startEndVisit();
+ thePrinter.endEndVisit();
+}
+// </IsTokenizerLangSupportedIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <StemIterator>
+void PrinterVisitor::beginVisit ( const StemIterator& a) {
+ thePrinter.startBeginVisit("StemIterator", ++theId);
+ printCommons( &a, theId );
+ thePrinter.endBeginVisit( theId );
+}
+
+void PrinterVisitor::endVisit ( const StemIterator& ) {
+ thePrinter.startEndVisit();
+ thePrinter.endEndVisit();
+}
+// </StemIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <StripDiacriticsIterator>
+void PrinterVisitor::beginVisit ( const StripDiacriticsIterator& a) {
+ thePrinter.startBeginVisit("StripDiacriticsIterator", ++theId);
+ printCommons( &a, theId );
+ thePrinter.endBeginVisit( theId );
+}
+
+void PrinterVisitor::endVisit ( const StripDiacriticsIterator& ) {
+ thePrinter.startEndVisit();
+ thePrinter.endEndVisit();
+}
+// </StripDiacriticsIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <ThesaurusLookupIterator>
+void PrinterVisitor::beginVisit ( const ThesaurusLookupIterator& a) {
+ thePrinter.startBeginVisit("ThesaurusLookupIterator", ++theId);
+ printCommons( &a, theId );
+ thePrinter.endBeginVisit( theId );
+}
+
+void PrinterVisitor::endVisit ( const ThesaurusLookupIterator& ) {
+ thePrinter.startEndVisit();
+ thePrinter.endEndVisit();
+}
+// </ThesaurusLookupIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <TokenizeIterator>
+void PrinterVisitor::beginVisit ( const TokenizeIterator& a) {
+ thePrinter.startBeginVisit("TokenizeIterator", ++theId);
+ printCommons( &a, theId );
+ thePrinter.endBeginVisit( theId );
+}
+
+void PrinterVisitor::endVisit ( const TokenizeIterator& ) {
+ thePrinter.startEndVisit();
+ thePrinter.endEndVisit();
+}
+// </TokenizeIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <TokenizerPropertiesIterator>
+void PrinterVisitor::beginVisit ( const TokenizerPropertiesIterator& a) {
+ thePrinter.startBeginVisit("TokenizerPropertiesIterator", ++theId);
+ printCommons( &a, theId );
+ thePrinter.endBeginVisit( theId );
+}
+
+void PrinterVisitor::endVisit ( const TokenizerPropertiesIterator& ) {
+ thePrinter.startEndVisit();
+ thePrinter.endEndVisit();
+}
+// </TokenizerPropertiesIterator>
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+// <TokenizeStringIterator>
+void PrinterVisitor::beginVisit ( const TokenizeStringIterator& a) {
+ thePrinter.startBeginVisit("TokenizeStringIterator", ++theId);
+ printCommons( &a, theId );
+ thePrinter.endBeginVisit( theId );
+}
+
+void PrinterVisitor::endVisit ( const TokenizeStringIterator& ) {
+ thePrinter.startEndVisit();
+ thePrinter.endEndVisit();
+}
+// </TokenizeStringIterator>
+
+#endif
// <FunctionNameIterator>
void PrinterVisitor::beginVisit ( const FunctionNameIterator& a) {
@@ -2674,6 +2870,20 @@
// </LeastCommonAncestor>
+// <FnPathIterator>
+void PrinterVisitor::beginVisit ( const FnPathIterator& a) {
+ thePrinter.startBeginVisit("FnPathIterator", ++theId);
+ printCommons( &a, theId );
+ thePrinter.endBeginVisit( theId );
+}
+
+void PrinterVisitor::endVisit ( const FnPathIterator& ) {
+ thePrinter.startEndVisit();
+ thePrinter.endEndVisit();
+}
+// </FnPathIterator>
+
+
// <AbsIterator>
void PrinterVisitor::beginVisit ( const AbsIterator& a) {
thePrinter.startBeginVisit("AbsIterator", ++theId);
@@ -2772,18 +2982,18 @@
// </FormatIntegerIterator>
-// <FnParseXmlFragmentIterator>
-void PrinterVisitor::beginVisit ( const FnParseXmlFragmentIterator& a) {
- thePrinter.startBeginVisit("FnParseXmlFragmentIterator", ++theId);
+// <FnZorbaParseXmlFragmentIterator>
+void PrinterVisitor::beginVisit ( const FnZorbaParseXmlFragmentIterator& a) {
+ thePrinter.startBeginVisit("FnZorbaParseXmlFragmentIterator", ++theId);
printCommons( &a, theId );
thePrinter.endBeginVisit( theId );
}
-void PrinterVisitor::endVisit ( const FnParseXmlFragmentIterator& ) {
+void PrinterVisitor::endVisit ( const FnZorbaParseXmlFragmentIterator& ) {
thePrinter.startEndVisit();
thePrinter.endEndVisit();
}
-// </FnParseXmlFragmentIterator>
+// </FnZorbaParseXmlFragmentIterator>
// <FnParseXmlIterator>
=== modified file 'src/runtime/visitors/pregenerated/printer_visitor.h'
--- src/runtime/visitors/pregenerated/printer_visitor.h 2012-04-24 12:39:38 +0000
+++ src/runtime/visitors/pregenerated/printer_visitor.h 2012-05-16 22:51:21 +0000
@@ -292,6 +292,71 @@
void beginVisit( const FnPutIterator& );
void endVisit ( const FnPutIterator& );
+#ifndef ZORBA_NO_FULL_TEXT
+ void beginVisit( const CurrentLangIterator& );
+ void endVisit ( const CurrentLangIterator& );
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+ void beginVisit( const HostLangIterator& );
+ void endVisit ( const HostLangIterator& );
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+ void beginVisit( const IsStemLangSupportedIterator& );
+ void endVisit ( const IsStemLangSupportedIterator& );
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+ void beginVisit( const IsStopWordIterator& );
+ void endVisit ( const IsStopWordIterator& );
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+ void beginVisit( const IsStopWordLangSupportedIterator& );
+ void endVisit ( const IsStopWordLangSupportedIterator& );
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+ void beginVisit( const IsThesaurusLangSupportedIterator& );
+ void endVisit ( const IsThesaurusLangSupportedIterator& );
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+ void beginVisit( const IsTokenizerLangSupportedIterator& );
+ void endVisit ( const IsTokenizerLangSupportedIterator& );
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+ void beginVisit( const StemIterator& );
+ void endVisit ( const StemIterator& );
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+ void beginVisit( const StripDiacriticsIterator& );
+ void endVisit ( const StripDiacriticsIterator& );
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+ void beginVisit( const ThesaurusLookupIterator& );
+ void endVisit ( const ThesaurusLookupIterator& );
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+ void beginVisit( const TokenizeIterator& );
+ void endVisit ( const TokenizeIterator& );
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+ void beginVisit( const TokenizerPropertiesIterator& );
+ void endVisit ( const TokenizerPropertiesIterator& );
+#endif
+
+#ifndef ZORBA_NO_FULL_TEXT
+ void beginVisit( const TokenizeStringIterator& );
+ void endVisit ( const TokenizeStringIterator& );
+#endif
+
void beginVisit( const FunctionNameIterator& );
void endVisit ( const FunctionNameIterator& );
@@ -598,6 +663,9 @@
void beginVisit( const LeastCommonAncestor& );
void endVisit ( const LeastCommonAncestor& );
+ void beginVisit( const FnPathIterator& );
+ void endVisit ( const FnPathIterator& );
+
void beginVisit( const AbsIterator& );
void endVisit ( const AbsIterator& );
@@ -619,8 +687,8 @@
void beginVisit( const FormatIntegerIterator& );
void endVisit ( const FormatIntegerIterator& );
- void beginVisit( const FnParseXmlFragmentIterator& );
- void endVisit ( const FnParseXmlFragmentIterator& );
+ void beginVisit( const FnZorbaParseXmlFragmentIterator& );
+ void endVisit ( const FnZorbaParseXmlFragmentIterator& );
void beginVisit( const FnParseXmlIterator& );
void endVisit ( const FnParseXmlIterator& );
=== modified file 'src/store/naive/atomic_items.cpp'
--- src/store/naive/atomic_items.cpp 2012-04-25 10:27:08 +0000
+++ src/store/naive/atomic_items.cpp 2012-05-16 22:51:21 +0000
@@ -1657,10 +1657,13 @@
{
typedef NaiveFTTokenIterator::container_type tokens_t;
unique_ptr<tokens_t> tokens( new tokens_t );
+ AtomicItemTokenizerCallback callback( *tokens );
- Tokenizer::ptr t( provider.getTokenizer( lang, numbers ) );
- AtomicItemTokenizerCallback cb( *t, lang, *tokens );
- cb.tokenize( theValue.data(), theValue.size(), wildcards );
+ Tokenizer::ptr tokenizer;
+ if ( provider.getTokenizer( lang, &numbers, &tokenizer ) )
+ tokenizer->tokenize_string(
+ theValue.data(), theValue.size(), lang, wildcards, callback
+ );
return FTTokenIterator_t( new NaiveFTTokenIterator( tokens.release() ) );
}
@@ -3503,11 +3506,11 @@
lStream.seekg(0, std::ios::end);
std::streampos len = lStream.tellg();
lStream.seekg(0, std::ios::beg);
- if (len < 0)
+ if (len < std::streampos(0))
{
throw ZORBA_EXCEPTION( zerr::ZOSE0003_STREAM_READ_FAILURE );
}
- if (len == 0)
+ if (len == std::streampos(0))
{
return;
}
@@ -3603,25 +3606,22 @@
********************************************************************************/
AtomicItemTokenizerCallback::AtomicItemTokenizerCallback(
- Tokenizer &tokenizer,
- locale::iso639_1::type lang,
container_type &tokens
) :
- tokenizer_( tokenizer ),
- lang_( lang ),
tokens_( tokens )
{
}
-void AtomicItemTokenizerCallback::operator()(
+void AtomicItemTokenizerCallback::token(
char const *utf8_s,
size_type utf8_len,
+ iso639_1::type lang,
size_type token_no,
size_type sent_no,
size_type para_no,
- void*
+ Item const*
) {
- FTToken const t( utf8_s, utf8_len, token_no, lang_ );
+ FTToken const t( utf8_s, utf8_len, token_no, lang );
tokens_.push_back( t );
}
=== modified file 'src/store/naive/atomic_items.h'
--- src/store/naive/atomic_items.h 2012-04-26 20:54:34 +0000
+++ src/store/naive/atomic_items.h 2012-05-16 22:51:21 +0000
@@ -1461,7 +1461,7 @@
xs_integer getIntegerValue() const { return theValue; }
- xs_long getLongValue() const;
+ xs_long getLongValue() const;
xs_unsignedInt getUnsignedIntValue() const;
@@ -2603,28 +2603,15 @@
public:
typedef FTTokenStore::container_type container_type;
- AtomicItemTokenizerCallback(
- Tokenizer &tokenizer,
- locale::iso639_1::type lang,
- container_type &tokens );
-
- void operator()(
- char const *utf8_s,
- size_type utf8_len,
- size_type token_no,
- size_type sent_no,
- size_type para_no,
- void* = 0 );
-
- void tokenize( char const *utf8_s, size_t len, bool wildcards = false )
- {
- tokenizer_.tokenize( utf8_s, len, lang_, wildcards, *this );
- }
+ AtomicItemTokenizerCallback( container_type &tokens );
+
+ // inherited
+ void token( char const *utf8_s, size_type utf8_len, locale::iso639_1::type,
+ size_type token_no, size_type sent_no, size_type para_no,
+ Item const* );
private:
- Tokenizer & tokenizer_;
- locale::iso639_1::type const lang_;
- container_type & tokens_;
+ container_type &tokens_;
};
#endif /* ZORBA_NO_FULL_TEXT */
=== modified file 'src/store/naive/loader_fast.cpp'
--- src/store/naive/loader_fast.cpp 2012-04-24 12:39:38 +0000
+++ src/store/naive/loader_fast.cpp 2012-05-16 22:51:21 +0000
@@ -494,10 +494,13 @@
// For each child, make this doc node its parent.
InternalNode::NodeVector& children = docNode->nodes();
+
numChildren = nodeStack.size() - firstChildPos - 1;
+
children.resize(numChildren);
numActualChildren = 0;
+
for (i = firstChildPos + 1; i < stackSize; ++i)
{
currChild = nodeStack[i];
=== modified file 'src/store/naive/node_items.cpp'
--- src/store/naive/node_items.cpp 2012-04-24 12:39:38 +0000
+++ src/store/naive/node_items.cpp 2012-05-16 22:51:21 +0000
@@ -21,6 +21,7 @@
#include <zorba/config.h>
#include <zorba/item.h>
+#include "api/unmarshaller.h"
#include "diagnostics/assert.h"
#include "diagnostics/xquery_diagnostics.h"
#include "zorbatypes/URI.h"
@@ -456,7 +457,7 @@
#ifndef NDEBUG
XmlNode::~XmlNode()
{
- NODE_TRACE1("Deleted " << store::StoreConsts::toString(getNodeKind()) << this);
+ STORE_TRACE1("Deleted " << store::StoreConsts::toString(getNodeKind()) << this);
}
#endif
@@ -1650,7 +1651,7 @@
:
InternalNode(store::StoreConsts::documentNode)
{
- NODE_TRACE1("Loaded doc node " << this);
+ STORE_TRACE1("Loaded doc node " << this);
}
@@ -1666,7 +1667,7 @@
theBaseUri(baseUri),
theDocUri(docUri)
{
- NODE_TRACE1("{\nConstructing doc node " << this << " tree = "
+ STORE_TRACE1("{\nConstructing doc node " << this << " tree = "
<< getTree()->getId() << ":" << getTree()
<< " doc uri = " << docUri);
}
@@ -1721,8 +1722,8 @@
throw;
}
- NODE_TRACE1("}");
- NODE_TRACE1("Copied doc node " << this << " to node " << copyNode);
+ STORE_TRACE1("}");
+ STORE_TRACE1("Copied doc node " << this << " to node " << copyNode);
return copyNode;
}
@@ -1891,7 +1892,7 @@
theNodes.resize(numAttributes);
}
- NODE_TRACE1("Loaded elem node " << this << " name = " << theName->show()
+ STORE_TRACE1("Loaded elem node " << this << " name = " << theName->show()
<< " num bindings = " << numBindings << " num attributes = "
<< numAttributes << std::endl);
}
@@ -2017,7 +2018,7 @@
throw;
}
- NODE_TRACE1("Constructed element node " << this << " parent = "
+ STORE_TRACE1("Constructed element node " << this << " parent = "
<< std::hex << (parent ? (ulong)parent : 0) << " pos = " << pos
<< " tree = " << getTree()->getId() << ":" << getTree()
<< " ordpath = " << theOrdPath.show()
@@ -2358,7 +2359,7 @@
throw;
}
- NODE_TRACE1("Copied elem node " << this << " to node " << copyNode
+ STORE_TRACE1("Copied elem node " << this << " to node " << copyNode
<< " name = " << theName->getStringValue() << " parent = "
<< (parent ? parent : 0x0)
<< " pos = " << pos << " copy mode = " << copymode.toString());
@@ -3326,7 +3327,7 @@
if (qn->isBaseUri())
theFlags |= IsBaseUri;
- NODE_TRACE1("Loaded attr node " << this << " name = " << theName->getStringValue());
+ STORE_TRACE1("Loaded attr node " << this << " name = " << theName->getStringValue());
}
@@ -3449,7 +3450,7 @@
throw;
}
- NODE_TRACE1("Constructed attribute node " << this << " parent = "
+ STORE_TRACE1("Constructed attribute node " << this << " parent = "
<< std::hex << (parent ? (ulong)parent : 0) << " pos = " << pos
<< " tree = " << getTree()->getId() << ":" << getTree()
<< " ordpath = " << theOrdPath.show()
@@ -3532,7 +3533,7 @@
throw;
}
- NODE_TRACE1("Copied attribute node " << this << " to node " << copyNode
+ STORE_TRACE1("Copied attribute node " << this << " to node " << copyNode
<< " name = " << theName->show() << " parent = "
<< std::hex << (parent ? (ulong)parent : 0) << " pos = " << pos
<< " copy mode = " << copymode.toString());
@@ -3788,7 +3789,7 @@
{
setText(content);
- NODE_TRACE1("Loaded text node " << this << " content = " << getText());
+ STORE_TRACE1("Loaded text node " << this << " content = " << getText());
}
@@ -3833,13 +3834,13 @@
}
#ifdef TEXT_ORDPATH
- NODE_TRACE1("Constructed text node " << this << " parent = "
+ STORE_TRACE1("Constructed text node " << this << " parent = "
<< std::hex << (parent ? (ulong)parent : 0) << " pos = " << pos
<< " tree = " << getTree()->getId() << ":" << getTree()
<< " ordpath = " << theOrdPath.show()
<< " content = " << getText());
#else
- NODE_TRACE1("Constructed text node " << this << " parent = "
+ STORE_TRACE1("Constructed text node " << this << " parent = "
<< std::hex << (parent ? (ulong)parent : 0) << " pos = " << pos
<< " tree = " << getTree()->getId() << ":" << getTree()
<< " content = " << getText());
@@ -3894,12 +3895,12 @@
p->insertChild(this, p->numChildren());
#ifdef TEXT_ORDPATH
- NODE_TRACE1("Constructed text node " << this << " parent = "
+ STORE_TRACE1("Constructed text node " << this << " parent = "
<< std::hex << (parent ? (ulong)parent : 0)
<< " ordpath = " << theOrdPath.show()
<< " content = " << getValue()->getStringValue());
#else
- NODE_TRACE1("Constructed text node " << this << " parent = "
+ STORE_TRACE1("Constructed text node " << this << " parent = "
<< std::hex << (parent ? (ulong)parent : 0)
<< " content = " << getValue()->getStringValue());
#endif
@@ -4011,7 +4012,7 @@
throw;
}
- NODE_TRACE1("Copied text node " << this << " to node " << copyNode
+ STORE_TRACE1("Copied text node " << this << " to node " << copyNode
<< " parent = " << std::hex << (parent ? (ulong)parent : 0)
<< " pos = " << pos);
@@ -4496,7 +4497,7 @@
theName = qnpool.insert(zstring(), zstring(), theTarget);
- NODE_TRACE1("Loaded pi node " << this << " target = " << theTarget
+ STORE_TRACE1("Loaded pi node " << this << " target = " << theTarget
<< std::endl);
}
@@ -4529,7 +4530,7 @@
parent->insertChild(this, pos);
}
- NODE_TRACE1("Constructed pi node " << this << " parent = "
+ STORE_TRACE1("Constructed pi node " << this << " parent = "
<< std::hex << (parent ? (ulong)parent : 0) << " pos = " << pos
<< " tree = " << getTree()->getId() << ":" << getTree()
<< " ordpath = " << theOrdPath.show() << " target = " << theTarget);
@@ -4578,7 +4579,7 @@
throw;
}
- NODE_TRACE1("Copied pi node " << this << " to node " << copyNode
+ STORE_TRACE1("Copied pi node " << this << " to node " << copyNode
<< " parent = " << std::hex << (parent ? (ulong)parent : 0)
<< " pos = " << pos);
@@ -4636,7 +4637,7 @@
{
theContent.take(content);
- NODE_TRACE1("Loaded comment node " << this << " content = " << theContent);
+ STORE_TRACE1("Loaded comment node " << this << " content = " << theContent);
}
@@ -4662,7 +4663,7 @@
parent->insertChild(this, pos);
}
- NODE_TRACE1("Constructed comment node " << this << " parent = "
+ STORE_TRACE1("Constructed comment node " << this << " parent = "
<< std::hex << (parent ? (ulong)parent : 0) << " pos = " << pos
<< " tree = " << getTree()->getId() << ":" << getTree()
<< " ordpath = " << theOrdPath.show() << " content = "
@@ -4709,7 +4710,7 @@
throw;
}
- NODE_TRACE1("Copied coment node " << this << " to node " << copyNode
+ STORE_TRACE1("Copied coment node " << this << " to node " << copyNode
<< " parent = " << std::hex << (parent ? (ulong)parent : 0)
<< " pos = " << pos);
@@ -4761,108 +4762,57 @@
******************************************************************************/
XmlNodeTokenizerCallback::XmlNodeTokenizerCallback(
- TokenizerProvider const &provider,
- Tokenizer::Numbers &numbers,
- iso639_1::type lang,
FTTokenStore &token_store
) :
- provider_( provider ),
- numbers_( numbers ),
token_store_( &token_store ),
tokens_( token_store.getDocumentTokens() )
{
- push_lang( lang );
}
XmlNodeTokenizerCallback::XmlNodeTokenizerCallback(
- TokenizerProvider const &provider,
- Tokenizer::Numbers &numbers,
- iso639_1::type lang,
container_type &tokens
) :
- provider_( provider ),
- numbers_( numbers ),
- token_store_( NULL ),
+ token_store_( nullptr ),
tokens_( tokens )
{
- push_lang( lang );
-}
-
-
-XmlNodeTokenizerCallback::~XmlNodeTokenizerCallback()
-{
- while ( !tokenizer_stack_.empty() )
- ztd::pop_stack( tokenizer_stack_ )->destroy();
-}
-
-
-inline XmlNodeTokenizerCallback::begin_type
-XmlNodeTokenizerCallback::beginTokenization() const
-{
- return token_store_->getDocumentTokens().size();
-}
-
-
-inline void XmlNodeTokenizerCallback::endTokenization(
- XmlNode const *node,
- XmlNodeTokenizerCallback::begin_type begin )
-{
- token_store_->putRange(node, begin, token_store_->getDocumentTokens().size());
-}
-
-
-void XmlNodeTokenizerCallback::pop_lang()
-{
- lang_stack_.pop();
- ztd::pop_stack( tokenizer_stack_ )->destroy();
-}
-
-
-void XmlNodeTokenizerCallback::push_lang( iso639_1::type lang )
-{
- lang_stack_.push( lang );
- Tokenizer::ptr t( provider_.getTokenizer( lang, numbers_ ) );
- ZORBA_ASSERT( t.get() );
- tokenizer_stack_.push( t.get() );
- t.release();
+}
+
+
+void XmlNodeTokenizerCallback::item( Item const &api_item, bool entering ) {
+ if ( token_store_ ) {
+ store::Item const *const item = Unmarshaller::getInternalItem( api_item );
+ if ( entering ) {
+ push_item( item );
+ range_stack_.push( token_store_->getDocumentTokens().size() );
+ } else {
+ pop_item();
+ token_store_->putRange(
+ item,
+ ztd::pop_stack( range_stack_ ),
+ token_store_->getDocumentTokens().size()
+ );
+ }
+ }
}
void XmlNodeTokenizerCallback::
-operator()( char const *utf8_s, size_type utf8_len, size_type pos,
- size_type sent, size_type para, void *payload )
+token( char const *utf8_s, size_type utf8_len, iso639_1::type lang,
+ size_type pos, size_type sent, size_type para, Item const *api_item )
{
- store::Item const *const item = static_cast<store::Item*>( payload );
- FTToken t( utf8_s, utf8_len, pos, sent, para, item, get_lang() );
+ store::Item const *const item = Unmarshaller::getInternalItem( *api_item );
+ FTToken t( utf8_s, utf8_len, pos, sent, para, item, lang );
tokens_.push_back( t );
}
-inline void XmlNodeTokenizerCallback::tokenize( char const *utf8_s,
- size_t len )
-{
- tokenizer().tokenize(
- utf8_s, len, get_lang(), false, *this,
- element_stack_.empty() ? NULL : static_cast<void*>( get_element() )
- );
-}
-
-
void XmlNode::tokenize( XmlNodeTokenizerCallback& )
{
// do nothing
}
-void AttributeNode::tokenize( XmlNodeTokenizerCallback &cb )
-{
- zstring text;
- getStringValue2( text );
- cb.tokenize( text.data(), text.size() );
-}
-
-
FTTokenIterator_t
AttributeNode::getTokens( TokenizerProvider const &provider,
Tokenizer::Numbers &numbers, iso639_1::type lang,
@@ -4875,62 +4825,21 @@
return FTTokenIterator_t(
new NaiveFTTokenIterator( *tokens, 0, tokens->size() )
);
+
FTTokenStore::container_type att_tokens;
- XmlNodeTokenizerCallback cb( provider, numbers, lang, att_tokens );
- const_cast<AttributeNode*>( this )->tokenize( cb );
- token_store.putAttr( this, att_tokens );
- }
-}
-
-
-void InternalNode::tokenize( XmlNodeTokenizerCallback& cb )
-{
- XmlNodeTokenizerCallback::begin_type const begin = cb.beginTokenization();
- for ( csize i = 0; i < numChildren(); ++i )
- getChild( i )->tokenize( cb );
- cb.endTokenization( this, begin );
-}
-
-
-void ElementNode::tokenize( XmlNodeTokenizerCallback& cb )
-{
- Tokenizer &tokenizer = cb.tokenizer();
-
- zorba::Item element_name;
- if ( tokenizer.trace_options() )
- element_name = getNodeName();
-
- if ( tokenizer.trace_options() & Tokenizer::trace_begin )
- tokenizer.element( element_name, Tokenizer::trace_begin );
- else if ( !tokenizer.trace_options() )
- ++tokenizer.numbers().para;
-
- //
- // See if this XML element has an xml:lang attribute: if so, switch to that
- // language.
- //
- bool pushed_lang = false;
- for ( ulong i = 0; i < numAttrs(); ++i ) {
- AttributeNode *const at = getAttr( i );
- Item const *const name = at->getNodeName();
- if ( name->getLocalName() == "lang" && name->getNamespace() == XML_NS ) {
- cb.push_lang( locale::find_lang( at->getStringValue().c_str() ) );
- pushed_lang = true;
- break;
+ XmlNodeTokenizerCallback callback( att_tokens );
+
+ zorba::Item const api_attr( this );
+ Tokenizer::ptr tokenizer;
+ if ( provider.getTokenizer( lang, &numbers, &tokenizer ) ) {
+ tokenizer->tokenize_node( api_attr, lang, callback );
+ token_store.putAttr( this, att_tokens );
}
}
-
- cb.push_element( this );
- InternalNode::tokenize( cb );
- cb.pop_element();
- if ( pushed_lang )
- cb.pop_lang();
-
- if ( tokenizer.trace_options() & Tokenizer::trace_end )
- tokenizer.element( element_name, Tokenizer::trace_end );
}
+#if 0
void TextNode::tokenize( XmlNodeTokenizerCallback &cb )
{
const zstring* text;
@@ -4986,6 +4895,7 @@
cb.tokenize( text->data(), text->size() );
cb.endTokenization( this, begin );
}
+#endif
FTTokenIterator_t
@@ -4998,8 +4908,11 @@
if ( tokens.empty() )
{
- XmlNodeTokenizerCallback cb( provider, numbers, lang, token_store );
- getRoot()->tokenize( cb );
+ zorba::Item const api_root( getRoot() );
+ XmlNodeTokenizerCallback callback( token_store );
+ Tokenizer::ptr tokenizer;
+ if ( provider.getTokenizer( lang, &numbers, &tokenizer ) )
+ tokenizer->tokenize_node( api_root, lang, callback );
}
FTTokenStore::range_type const &r = token_store.getRange( this );
=== modified file 'src/store/naive/node_items.h'
--- src/store/naive/node_items.h 2012-04-24 12:39:38 +0000
+++ src/store/naive/node_items.h 2012-05-16 22:51:21 +0000
@@ -107,28 +107,6 @@
<< store::StoreConsts::toString(getNodeKind()))
-#ifndef NDEBUG
-
-#define NODE_TRACE(level, msg) \
-{ \
- if (level <= GET_STORE().getTraceLevel()) \
- std::cout << msg << std::endl; \
-}
-
-#define NODE_TRACE1(msg) NODE_TRACE(1, msg);
-#define NODE_TRACE2(msg) NODE_TRACE(2, msg);
-#define NODE_TRACE3(msg) NODE_TRACE(3, msg);
-
-#else
-
-#define NODE_TRACE(msg)
-#define NODE_TRACE1(msg)
-#define NODE_TRACE2(msg)
-#define NODE_TRACE3(msg)
-
-#endif
-
-
/*******************************************************************************
theRefCount : It is the sum of theRefCounts of all the nodes belonging to
@@ -884,10 +862,6 @@
const OrdPath* getFirstChildOrdPathAfter(csize pos) const;
const OrdPath* getFirstChildOrdPathBefore(csize pos) const;
-
-#ifndef ZORBA_NO_FULL_TEXT
- void tokenize( XmlNodeTokenizerCallback& );
-#endif /* ZORBA_NO_FULL_TEXT */
};
@@ -1147,10 +1121,6 @@
zstring& absUri,
zstring& relUri);
-#ifndef ZORBA_NO_FULL_TEXT
- void tokenize( XmlNodeTokenizerCallback& );
-#endif /* ZORBA_NO_FULL_TEXT */
-
private:
//disable default copy constructor
ElementNode(const ElementNode& src);
@@ -1264,7 +1234,8 @@
#ifndef ZORBA_NO_FULL_TEXT
FTTokenIterator_t getTokens( TokenizerProvider const&, Tokenizer::Numbers&,
- locale::iso639_1::type, bool = false ) const;
+ locale::iso639_1::type,
+ bool wildcards = false ) const;
#endif /* ZORBA_NO_FULL_TEXT */
protected:
@@ -1279,10 +1250,6 @@
{
return *reinterpret_cast<ItemVector*>(theTypedValue.getp());
}
-
-#ifndef ZORBA_NO_FULL_TEXT
- void tokenize( XmlNodeTokenizerCallback& );
-#endif
store::Iterator_t getChildren() const;
};
@@ -1441,10 +1408,6 @@
void setValue(store::Item_t& val) { theContent.setValue(val); }
void setValue(store::Item* val) { theContent.setValue(val); }
-
-#ifndef ZORBA_NO_FULL_TEXT
- void tokenize( XmlNodeTokenizerCallback& );
-#endif /* ZORBA_NO_FULL_TEXT */
store::Iterator_t getChildren() const;
};
@@ -1680,59 +1643,26 @@
{
public:
typedef FTTokenStore::container_type container_type;
- typedef FTTokenStore::size_type begin_type;
-
- XmlNodeTokenizerCallback( TokenizerProvider const &provider,
- Tokenizer::Numbers &numbers,
- locale::iso639_1::type lang,
- FTTokenStore &token_store );
-
- XmlNodeTokenizerCallback( TokenizerProvider const &provider,
- Tokenizer::Numbers &numbers,
- locale::iso639_1::type lang,
- container_type &tokens );
-
- ~XmlNodeTokenizerCallback();
-
- begin_type beginTokenization() const;
-
- void endTokenization( XmlNode const*, begin_type );
-
- void push_element( ElementNode *element ) { element_stack_.push( element ); }
-
- void pop_element() { element_stack_.pop(); }
-
- void push_lang( locale::iso639_1::type lang );
-
- void pop_lang();
-
- void tokenize( char const *utf8_s, size_t len );
-
- Tokenizer& tokenizer() const { return *tokenizer_stack_.top(); }
+
+ XmlNodeTokenizerCallback( FTTokenStore &token_store );
+ XmlNodeTokenizerCallback( container_type &tokens );
// inherited
- void operator()( char const *utf8_s, size_type utf8_len,
- size_type pos, size_type sent, size_type para, void* );
+ void item( Item const&, bool );
+ void token( char const *utf8_s, size_type utf8_len, locale::iso639_1::type,
+ size_type pos, size_type sent, size_type para, Item const* );
private:
- typedef std::stack<ElementNode*> element_stack_t;
- typedef std::stack<locale::iso639_1::type> lang_stack_t;
- typedef std::stack<Tokenizer*> tokenizer_stack_t;
-
- ElementNode* get_element() const {
- return element_stack_.top();
- }
-
- locale::iso639_1::type get_lang() const {
- return lang_stack_.top();
- }
-
- TokenizerProvider const &provider_;
- Tokenizer::Numbers &numbers_;
+ typedef std::stack<store::Item const*> item_stack_t;
+ typedef std::stack<FTTokenStore::size_type> range_begin_stack_t;
+
+ store::Item const* get_item() const { return item_stack_.top(); }
+ void push_item( store::Item const *item ) { item_stack_.push( item ); }
+ void pop_item() { item_stack_.pop(); }
+
FTTokenStore *token_store_;
container_type &tokens_;
- element_stack_t element_stack_;
- lang_stack_t lang_stack_;
- tokenizer_stack_t tokenizer_stack_;
+ item_stack_t item_stack_;
+ range_begin_stack_t range_stack_;
};
#endif /* ZORBA_NO_FULL_TEXT */
=== modified file 'src/store/naive/pul_primitives.cpp'
--- src/store/naive/pul_primitives.cpp 2012-04-24 12:39:38 +0000
+++ src/store/naive/pul_primitives.cpp 2012-05-16 22:51:21 +0000
@@ -35,6 +35,7 @@
#include "store/api/validator.h"
#include "diagnostics/xquery_diagnostics.h"
+#include "diagnostics/util_macros.h"
namespace zorba {
@@ -946,42 +947,36 @@
void UpdDeleteCollection::apply()
{
theCollection = GET_STORE().getCollection(theName, theDynamicCollection);
+
if (theCollection == NULL)
return;//If two delete collection are issued in the same snapshot is a noop
+
Collection* collection = static_cast<Collection*>(theCollection.getp());
std::vector<store::Index*> indexes;
collection->getIndexes(indexes);
if (!indexes.empty())
- throw XQUERY_EXCEPTION(
- zerr::ZDDY0013_COLLECTION_BAD_DESTROY_INDEXES,
- ERROR_PARAMS( collection->getName()->getStringValue() ),
- ERROR_LOC( theLoc )
- );
+ RAISE_ERROR(zerr::ZDDY0013_COLLECTION_BAD_DESTROY_INDEXES, theLoc,
+ ERROR_PARAMS(collection->getName()->getStringValue()));
std::vector<store::IC*> activeICs;
collection->getActiveICs(activeICs);
if (!activeICs.empty())
- throw XQUERY_EXCEPTION(
- zerr::ZDDY0014_COLLECTION_BAD_DESTROY_ICS,
- ERROR_PARAMS( collection->getName()->getStringValue() ),
- ERROR_LOC( theLoc )
- );
+ RAISE_ERROR(zerr::ZDDY0014_COLLECTION_BAD_DESTROY_ICS, theLoc,
+ ERROR_PARAMS(collection->getName()->getStringValue()));
uint64_t size;
- try {
+ try
+ {
size = to_xs_unsignedLong(collection->size());
- } catch (std::range_error& e)
+ }
+ catch (std::range_error& e)
{
- throw ZORBA_EXCEPTION(
- zerr::ZSTR0060_RANGE_EXCEPTION,
- ERROR_PARAMS(
- BUILD_STRING("collection too big ("
- << e.what() << "; " << theName << ")")
- )
- );
+ throw ZORBA_EXCEPTION(zerr::ZSTR0060_RANGE_EXCEPTION,
+ ERROR_PARAMS(BUILD_STRING("collection too big ("
+ << e.what() << "; " << theName << ")")));
}
for (uint64_t i = 0; i < size; ++i)
@@ -989,11 +984,10 @@
XmlNode* root = static_cast<XmlNode*>(collection->nodeAt(xs_integer(i)).getp());
XmlTree* tree = root->getTree();
if (tree->getRefCount() > 1)
- throw XQUERY_EXCEPTION(
- zerr::ZDDY0015_COLLECTION_BAD_DESTROY_NODES,
- ERROR_PARAMS( collection->getName()->getStringValue() ),
- ERROR_LOC( theLoc )
- );
+ {
+ RAISE_ERROR(zerr::ZDDY0015_COLLECTION_BAD_DESTROY_NODES, theLoc,
+ ERROR_PARAMS(collection->getName()->getStringValue()));
+ }
}
GET_STORE().deleteCollection(theName, theDynamicCollection);
@@ -1013,13 +1007,13 @@
void UpdInsertIntoCollection::apply()
{
Collection* lColl = static_cast<Collection*>
- (GET_STORE().getCollection(theName, theDynamicCollection).getp());
+ (GET_STORE().getCollection(theName, theDynamicCollection).getp());
assert(lColl);
theIsApplied = true;
- std::size_t numNodes = theNodes.size();
- for (std::size_t i = 0; i < numNodes; ++i)
+ csize numNodes = theNodes.size();
+ for (csize i = 0; i < numNodes; ++i)
{
lColl->addNode(theNodes[i], xs_integer(-1));
++theNumApplied;
@@ -1030,7 +1024,7 @@
void UpdInsertIntoCollection::undo()
{
Collection* lColl = static_cast<Collection*>
- (GET_STORE().getCollection(theName, theDynamicCollection).getp());
+ (GET_STORE().getCollection(theName, theDynamicCollection).getp());
assert(lColl);
uint64_t lastPos;
@@ -1040,13 +1034,9 @@
}
catch (std::range_error& e)
{
- throw ZORBA_EXCEPTION(
- zerr::ZSTR0060_RANGE_EXCEPTION,
- ERROR_PARAMS(
- BUILD_STRING("collection too big ("
- << e.what() << "; " << theName << ")")
- )
- );
+ throw ZORBA_EXCEPTION(zerr::ZSTR0060_RANGE_EXCEPTION,
+ ERROR_PARAMS(BUILD_STRING("collection too big ("
+ << e.what() << "; " << theName << ")")));
}
for (long i = theNumApplied-1; i >= 0; --i)
@@ -1154,7 +1144,7 @@
void UpdInsertBeforeIntoCollection::apply()
{
Collection* lColl = static_cast<Collection*>
- (GET_STORE().getCollection(theName, theDynamicCollection).getp());
+ (GET_STORE().getCollection(theName, theDynamicCollection).getp());
assert(lColl);
if (!theNodes.empty())
@@ -1170,7 +1160,7 @@
void UpdInsertBeforeIntoCollection::undo()
{
Collection* lColl = static_cast<Collection*>
- (GET_STORE().getCollection(theName, theDynamicCollection).getp());
+ (GET_STORE().getCollection(theName, theDynamicCollection).getp());
assert(lColl);
ZORBA_ASSERT(theFirstNode == lColl->nodeAt(theFirstPos));
@@ -1184,7 +1174,7 @@
void UpdInsertAfterIntoCollection::apply()
{
Collection* lColl = static_cast<Collection*>
- (GET_STORE().getCollection(theName, theDynamicCollection).getp());
+ (GET_STORE().getCollection(theName, theDynamicCollection).getp());
assert(lColl);
if (!theNodes.empty())
@@ -1201,7 +1191,7 @@
void UpdInsertAfterIntoCollection::undo()
{
Collection* lColl = static_cast<Collection*>
- (GET_STORE().getCollection(theName, theDynamicCollection).getp());
+ (GET_STORE().getCollection(theName, theDynamicCollection).getp());
assert(lColl);
ZORBA_ASSERT(theFirstNode == lColl->nodeAt(theFirstPos));
@@ -1236,13 +1226,13 @@
);
}
- std::size_t numNodes = theNodes.size();
+ csize numNodes = theNodes.size();
bool isLast = theIsLast;
if (theIsLast)
{
- for (std::size_t i = numNodes; i > 0; --i)
+ for (csize i = numNodes; i > 0; --i)
{
if (theNodes[i-1] != lColl->nodeAt(xs_integer(size - i)))
{
@@ -1270,7 +1260,7 @@
void UpdDeleteNodesFromCollection::undo()
{
Collection* lColl = static_cast<Collection*>
- (GET_STORE().getCollection(theName, theDynamicCollection).getp());
+ (GET_STORE().getCollection(theName, theDynamicCollection).getp());
assert(lColl);
for (std::size_t i = 0; i < theNumApplied; ++i)
@@ -1289,7 +1279,7 @@
void UpdTruncateCollection::apply()
{
Collection* lColl = static_cast<Collection*>
- (GET_STORE().getCollection(theName, theDynamicCollection).getp());
+ (GET_STORE().getCollection(theName, theDynamicCollection).getp());
assert(lColl);
lColl->removeAll();
=== modified file 'src/store/naive/simple_index_value.cpp'
--- src/store/naive/simple_index_value.cpp 2012-04-24 12:39:38 +0000
+++ src/store/naive/simple_index_value.cpp 2012-05-16 22:51:21 +0000
@@ -312,6 +312,15 @@
********************************************************************************/
ValueHashIndex::~ValueHashIndex()
{
+ clear();
+}
+
+
+/*******************************************************************************
+
+********************************************************************************/
+void ValueHashIndex::clear()
+{
IndexMap::iterator ite = theMap.begin();
IndexMap::iterator end = theMap.end();
@@ -323,14 +332,7 @@
delete (*ite).first;
delete (*ite).second;
}
-}
-
-
-/*******************************************************************************
-
-********************************************************************************/
-void ValueHashIndex::clear()
-{
+
theMap.clear();
}
@@ -366,9 +368,9 @@
ERROR_PARAMS(key->toString(), theQname->getStringValue()));
}
- ValueIndexValue* valueSet = NULL;
+ IndexMap::iterator pos = theMap.find(key);
- if (theMap.get(key, valueSet))
+ if (pos != theMap.end())
{
if (isUnique())
{
@@ -376,21 +378,20 @@
ERROR_PARAMS(theQname->getStringValue()));
}
- valueSet->resize(valueSet->size() + 1);
- (*valueSet)[valueSet->size()-1].transfer(value);
-
+ (*pos).second->transfer_back(value);
+ key = const_cast<store::IndexKey*>((*pos).first);
+
return true;
}
- valueSet = new ValueIndexValue(1);
+ ValueIndexValue* valueSet = new ValueIndexValue(1);
(*valueSet)[0].transfer(value);
//std::cout << "Index Entry Insert [" << key << ","
// << valueSet << "]" << std::endl;
- const store::IndexKey* key2 = key;
- theMap.insert(key2, valueSet);
- key = NULL; // ownership of the key obj passes to the index.
+ // Note: ownership of the key obj passes to the index.
+ theMap.insert(key, valueSet);
return false;
}
@@ -407,7 +408,7 @@
********************************************************************************/
bool ValueHashIndex::remove(
const store::IndexKey* key,
- store::Item_t& value,
+ const store::Item_t& value,
bool all)
{
if (key->size() != getNumColumns())
@@ -593,14 +594,7 @@
********************************************************************************/
ValueTreeIndex::~ValueTreeIndex()
{
- IndexMap::iterator ite = theMap.begin();
- IndexMap::iterator end = theMap.end();
-
- for (; ite != end; ++ite)
- {
- delete (*ite).first;
- delete (*ite).second;
- }
+ clear();
}
@@ -609,6 +603,15 @@
********************************************************************************/
void ValueTreeIndex::clear()
{
+ IndexMap::iterator ite = theMap.begin();
+ IndexMap::iterator end = theMap.end();
+
+ for (; ite != end; ++ite)
+ {
+ delete (*ite).first;
+ delete (*ite).second;
+ }
+
theMap.clear();
}
@@ -649,7 +652,7 @@
#if 0
std::cout << "inserting entry : [(";
- for (ulong i = 0; i < getNumColumns(); i++)
+ for (csize i = 0; i < getNumColumns(); i++)
{
if (key[i] != NULL)
std::cout << key[i]->getStringValue() << ", ";
@@ -672,14 +675,16 @@
}
pos->second->transfer_back(value);
+ key = const_cast<store::IndexKey*>(pos->first);
+
return true;
}
ValueIndexValue* valueSet = new ValueIndexValue(1);
(*valueSet)[0].transfer(value);
+ // Note: ownership of the key obj passes to the index.
theMap.insert(IndexMapPair(key, valueSet));
- key = NULL; // ownership of the key obj passes to the index.
return false;
}
@@ -690,7 +695,7 @@
********************************************************************************/
bool ValueTreeIndex::remove(
const store::IndexKey* key,
- store::Item_t& value,
+ const store::Item_t& value,
bool all)
{
if (key->size() != getNumColumns())
=== modified file 'src/store/naive/simple_index_value.h'
--- src/store/naive/simple_index_value.h 2012-04-24 12:39:38 +0000
+++ src/store/naive/simple_index_value.h 2012-05-16 22:51:21 +0000
@@ -88,11 +88,13 @@
public:
const XQPCollator* getCollator(ulong i) const;
+ virtual bool isTreeIndex() = 0;
+
virtual bool insert(store::IndexKey*& key, store::Item_t& item) = 0;
virtual bool remove(
const store::IndexKey* key,
- store::Item_t& item,
+ const store::Item_t& item,
bool all = false) = 0;
};
@@ -142,6 +144,8 @@
~ValueHashIndex();
public:
+ bool isTreeIndex() { return false; }
+
void clear();
ulong size() const;
@@ -150,7 +154,7 @@
bool insert(store::IndexKey*& key, store::Item_t& item);
- bool remove(const store::IndexKey* key, store::Item_t& item, bool all);
+ bool remove(const store::IndexKey* key, const store::Item_t& item, bool all);
};
@@ -227,6 +231,8 @@
~ValueTreeIndex();
public:
+ bool isTreeIndex() { return true; }
+
void clear();
ulong size() const;
@@ -235,7 +241,7 @@
bool insert(store::IndexKey*& key, store::Item_t& item);
- bool remove(const store::IndexKey* key, store::Item_t& item, bool all = false);
+ bool remove(const store::IndexKey* key, const store::Item_t& item, bool all = false);
};
=== modified file 'src/store/naive/simple_pul.cpp'
--- src/store/naive/simple_pul.cpp 2012-04-24 12:39:38 +0000
+++ src/store/naive/simple_pul.cpp 2012-05-16 22:51:21 +0000
@@ -58,28 +58,6 @@
/*******************************************************************************
********************************************************************************/
-void cleanIndexDeltas(std::vector<store::IndexDelta>& deltas)
-{
- std::vector<store::IndexDelta>::iterator ite = deltas.begin();
- std::vector<store::IndexDelta>::iterator end = deltas.end();
-
- for (; ite != end; ++ite)
- {
- store::IndexDelta::iterator ite2 = (*ite).begin();
- store::IndexDelta::iterator end2 = (*ite).end();
-
- for (; ite2 != end2; ++ite2)
- {
- if ((*ite2).second)
- delete (*ite2).second;
- }
- }
-}
-
-
-/*******************************************************************************
-
-********************************************************************************/
void applyList(std::vector<UpdatePrimitive*>& aVector)
{
std::vector<UpdatePrimitive*>::iterator iter = aVector.begin();
@@ -97,9 +75,9 @@
********************************************************************************/
void undoList(std::vector<UpdatePrimitive*>& list)
{
- ulong size = (ulong)list.size();
+ csize size = list.size();
- for (ulong i = size; i > 0; --i)
+ for (csize i = size; i > 0; --i)
{
if (list[i-1]->isApplied())
list[i-1]->undo();
@@ -150,14 +128,14 @@
cleanList(theValidationList);
- CollectionPulMap::iterator ite = theCollectionPuls.begin();
- CollectionPulMap::iterator end = theCollectionPuls.end();
+ CollectionPuls::iterator ite = theCollectionPuls.begin();
+ CollectionPuls::iterator end = theCollectionPuls.end();
for (; ite != end; ++ite)
{
- if ((*ite).second != NULL)
- delete (*ite).second;
+ delete (*ite);
}
+
cleanList(theICActivationList);
cleanList(theCreateDocumentList);
cleanList(theDeleteDocumentList);
@@ -197,49 +175,53 @@
if (collName == theLastCollection)
return theLastPul;
- return getCollectionPulByName(collName,collection->isDynamic());
+
+ return getCollectionPulByName(collName, collection->isDynamic());
}
else if (theNoCollectionPul != NULL)
{
return theNoCollectionPul;
}
- else if (theCollectionPuls[NULL] != NULL)
- {
- theNoCollectionPul = theCollectionPuls[NULL];
- return theNoCollectionPul;
- }
else
{
theNoCollectionPul = new CollectionPul(this, NULL);
- theCollectionPuls[NULL] = theNoCollectionPul;
+ theCollectionPuls.push_back(theNoCollectionPul);
+ theCollectionPulsMap[NULL] = theCollectionPuls.size() - 1;
return theNoCollectionPul;
}
}
-CollectionPul* PULImpl::getCollectionPulByName(const store::Item* name, bool dynamicCollection)
+
+CollectionPul* PULImpl::getCollectionPulByName(
+ const store::Item* name,
+ bool isDynamic)
{
const QNameItem* collName = static_cast<const QNameItem*>(name)->getNormalized();
- assert(!name->isNode());
+ assert(name->isAtomic());
// "name" is the name of a collection.
if (name == theLastCollection)
return theLastPul;
- CollectionPulMap::iterator ite = theCollectionPuls.find(collName);
+ CollectionPulMap::iterator ite = theCollectionPulsMap.find(collName);
theLastCollection = collName;
- if (ite != theCollectionPuls.end())
+ if (ite != theCollectionPulsMap.end())
{
- theLastPul = ite->second;
+ theLastPul = theCollectionPuls[ite->second];
}
else
{
Collection* collection = static_cast<Collection*>
- (GET_STORE().getCollection(collName,dynamicCollection).getp());
+ (GET_STORE().getCollection(collName, isDynamic).getp());
+
theLastPul = new CollectionPul(this, collection);
- theCollectionPuls[collName] = theLastPul;
+
+ theCollectionPuls.push_back(theLastPul);
+
+ theCollectionPulsMap[collName] = theCollectionPuls.size() - 1;
}
return theLastPul;
@@ -1107,17 +1089,17 @@
PULImpl* otherp = reinterpret_cast<PULImpl*>(other);
// Merge collection-specific primitives
- CollectionPulMap::iterator thisIte = theCollectionPuls.begin();
- CollectionPulMap::iterator thisEnd = theCollectionPuls.end();
- CollectionPulMap::iterator otherIte = otherp->theCollectionPuls.begin();
- CollectionPulMap::iterator otherEnd = otherp->theCollectionPuls.end();
+ CollectionPulMap::iterator thisIte = theCollectionPulsMap.begin();
+ CollectionPulMap::iterator thisEnd = theCollectionPulsMap.end();
+ CollectionPulMap::iterator otherIte = otherp->theCollectionPulsMap.begin();
+ CollectionPulMap::iterator otherEnd = otherp->theCollectionPulsMap.end();
while (thisIte != thisEnd && otherIte != otherEnd)
{
if (thisIte->first == otherIte->first)
{
- CollectionPul* thisPul = thisIte->second;
- CollectionPul* otherPul = otherIte->second;
+ CollectionPul* thisPul = theCollectionPuls[thisIte->second];
+ CollectionPul* otherPul = otherp->theCollectionPuls[otherIte->second];
// Merge XQUF primitives
mergeUpdateList(thisPul,
@@ -1186,18 +1168,27 @@
}
else
{
- theCollectionPuls[otherIte->first] = otherIte->second;
- otherIte->second->thePul = this;
- otherIte->second = NULL;
+ CollectionPul* otherPul = otherp->theCollectionPuls[otherIte->second];
+ otherp->theCollectionPuls[otherIte->second] = NULL;
+
+ theCollectionPuls.push_back(otherPul);
+ theCollectionPulsMap[otherIte->first] = theCollectionPuls.size() - 1;
+
+ otherPul->switchPul(this);
++otherIte;
}
}
while (otherIte != otherEnd)
{
- theCollectionPuls[otherIte->first] = otherIte->second;
- otherIte->second->switchPul(this);
- otherIte->second = NULL;
+ CollectionPul* otherPul = otherp->theCollectionPuls[otherIte->second];
+ otherp->theCollectionPuls[otherIte->second] = NULL;
+
+ theCollectionPuls.push_back(otherPul);
+ theCollectionPulsMap[otherIte->first] = theCollectionPuls.size() - 1;
+
+ otherPul->switchPul(this);
+
++otherIte;
}
@@ -1470,14 +1461,14 @@
********************************************************************************/
void PULImpl::checkTransformUpdates(const std::vector<store::Item*>& rootNodes) const
{
- ulong numRoots = (ulong)rootNodes.size();
+ csize numRoots = rootNodes.size();
- CollectionPulMap::const_iterator collIte = theCollectionPuls.begin();
- CollectionPulMap::const_iterator collEnd = theCollectionPuls.end();
+ std::vector<CollectionPul*>::const_iterator collIte = theCollectionPuls.begin();
+ std::vector<CollectionPul*>::const_iterator collEnd = theCollectionPuls.end();
for (; collIte != collEnd; ++collIte)
{
- CollectionPul* pul = collIte->second;
+ CollectionPul* pul = *collIte;
NodeToUpdatesMap::iterator it = pul->theNodeToUpdatesMap.begin();
NodeToUpdatesMap::iterator end = pul->theNodeToUpdatesMap.end();
@@ -1488,7 +1479,7 @@
bool found = false;
- for (ulong i = 0; i < numRoots; i++)
+ for (csize i = 0; i < numRoots; i++)
{
XmlNode* rootNode = reinterpret_cast<XmlNode*>(rootNodes[i]);
@@ -1533,12 +1524,12 @@
std::set<store::Collection*> collections;
std::set<store::Collection*> truncated_collections;
- CollectionPulMap::iterator collIte = theCollectionPuls.begin();
- CollectionPulMap::iterator collEnd = theCollectionPuls.end();
+ CollectionPuls::iterator collIte = theCollectionPuls.begin();
+ CollectionPuls::iterator collEnd = theCollectionPuls.end();
for (; collIte != collEnd; ++collIte)
{
- store::Collection* collection = store->getCollection(collIte->first);
+ store::Collection* collection = (*collIte)->theCollection;
// The collection may not be created yet.
if (collection == NULL)
@@ -1546,7 +1537,7 @@
collections.insert(collection);
- CollectionPul* pul = collIte->second;
+ CollectionPul* pul = *collIte;
if (pul->theTruncateCollectionList.size() > 0)
{
@@ -1570,6 +1561,7 @@
(pul->theInsertIntoCollectionList[i]);
csize numDocs = upd->numNodes();
+
for (csize j = 0; j < numDocs; ++j)
pul->theInsertedDocs.push_back(static_cast<XmlNode*>(upd->getNode(j)));
}
@@ -1582,6 +1574,7 @@
(pul->theDeleteFromCollectionList[i]);
csize numDocs = upd->numNodes();
+
for (csize j = 0; j < numDocs; ++j)
pul->theDeletedDocs.push_back(static_cast<XmlNode*>(upd->getNode(j)));
}
@@ -1689,8 +1682,8 @@
********************************************************************************/
void PULImpl::applyUpdates(bool inheritNSBindings)
{
- CollectionPulMap::iterator collIte = theCollectionPuls.begin();
- CollectionPulMap::iterator collEnd = theCollectionPuls.end();
+ CollectionPuls::iterator collIte = theCollectionPuls.begin();
+ CollectionPuls::iterator collEnd = theCollectionPuls.end();
theInheritNSBindings = inheritNSBindings;
@@ -1701,7 +1694,7 @@
// maintained incrementally w.r.t. updates in C.
for (; collIte != collEnd; ++collIte)
{
- CollectionPul* pul = collIte->second;
+ CollectionPul* pul = *collIte;
pul->applyUpdates();
}
@@ -1726,7 +1719,7 @@
// check integrity constraints for involved collections
for (collIte = theCollectionPuls.begin(); collIte != collEnd; ++collIte)
{
- CollectionPul* pul = collIte->second;
+ CollectionPul* pul = *collIte;
if (pul->theCollection != NULL)
{
@@ -1743,9 +1736,18 @@
// Apply delete-collection primitives
for (collIte = theCollectionPuls.begin(); collIte != collEnd; ++collIte)
{
- CollectionPul* pul = collIte->second;
+ CollectionPul* pul = *collIte;
applyList(pul->theDeleteCollectionList);
}
+
+ // Refresh each incrementally maintained index. We need to do this here
+ // because refreshIndices can raise an error (e.g. if the unique constraint
+ // of an index is violated).
+ for (collIte = theCollectionPuls.begin(); collIte != collEnd; ++collIte)
+ {
+ CollectionPul* pul = *collIte;
+ pul->refreshIndexes();
+ }
}
catch (...)
{
@@ -1753,10 +1755,10 @@
throw;
}
- //
+ // Perform actions that are not expected to raise any errors
for (collIte = theCollectionPuls.begin(); collIte != collEnd; ++collIte)
{
- CollectionPul* pul = collIte->second;
+ CollectionPul* pul = *collIte;
pul->finalizeUpdates();
}
@@ -1790,12 +1792,12 @@
{
undoList(theValidationList);
- CollectionPulMap::iterator collIte = theCollectionPuls.begin();
- CollectionPulMap::iterator collEnd = theCollectionPuls.end();
+ CollectionPuls::iterator collIte = theCollectionPuls.begin();
+ CollectionPuls::iterator collEnd = theCollectionPuls.end();
for (; collIte != collEnd; ++collIte)
{
- CollectionPul* pul = collIte->second;
+ CollectionPul* pul = *collIte;
undoList(pul->theDeleteCollectionList);
}
@@ -1807,7 +1809,7 @@
for (collIte = theCollectionPuls.begin(); collIte != collEnd; ++collIte)
{
- CollectionPul* pul = collIte->second;
+ CollectionPul* pul = *collIte;
pul->undoUpdates();
}
@@ -1851,10 +1853,7 @@
cleanList(theTruncateCollectionList);
cleanList(theDeleteCollectionList);
- cleanIndexDeltas(theBeforeIndexDeltas);
- cleanIndexDeltas(theAfterIndexDeltas);
- cleanIndexDeltas(theInsertedDocsIndexDeltas);
- cleanIndexDeltas(theDeletedDocsIndexDeltas);
+ cleanIndexDeltas();
}
@@ -1890,22 +1889,44 @@
}
}
-/*******************************************************************************
- The comparison function for sorting the entries of an IndexDelta by the doc node
-********************************************************************************/
-static bool cmp(const std::pair<store::Item_t, store::IndexKey*>& e1,
- const std::pair<store::Item_t, store::IndexKey*>& e2)
-{
- return e1.first.getp() < e2.first.getp();
-}
-
-
-/*******************************************************************************
- Compute the index contents on the modified docs, before any modifications
- are actually applied.
+
+/*******************************************************************************
+ For each incrementally-maintained index associated with this collection,
+ compute the index contents on the modified and deleted docs, before any
+ modifications are actually applied.
+
+ Note 1: If any docs are deleted, we have to remove from the before and after
+ deltas any entries for nodes belonging to the deleted docs. This is required
+ for the undo to work properly. For example, let E = [N, K] be an after-delta
+ entry, and let N be a node in a doc D that is going to be removed from the
+ collection. Then, during undo, the key pointer in E may be a dangling pointer.
+
+ Note 2: Given note 1, we actually have to compute the delete-docs deltas
+ *before* any modification are actually applied.
********************************************************************************/
void CollectionPul::computeIndexBeforeDeltas()
{
+ csize numIncrementalIndices = theIncrementalIndices.size();
+
+ if (numIncrementalIndices == 0)
+ return;
+
+ std::vector<XmlNode*>::const_iterator docIte = theDeletedDocs.begin();
+ std::vector<XmlNode*>::const_iterator docEnd = theDeletedDocs.end();
+
+ for (; docIte != docEnd; ++docIte)
+ {
+ theModifiedDocs.erase(*docIte);
+
+ for (csize i = 0; i < numIncrementalIndices; ++i)
+ {
+ store::IndexEntryCreator* docIndexer = theIndexEntryCreators[i].getp();
+ store::IndexDelta& indexDelta = theDeletedDocsIndexDeltas[i];
+
+ docIndexer->createIndexEntries((*docIte), indexDelta);
+ }
+ }
+
computeIndexDeltas(theBeforeIndexDeltas);
}
@@ -1917,21 +1938,19 @@
********************************************************************************/
void CollectionPul::computeIndexAfterDeltas()
{
+ csize numIncrementalIndices = theIncrementalIndices.size();
+
+ if (numIncrementalIndices == 0)
+ return;
+
computeIndexDeltas(theAfterIndexDeltas);
- csize numIncrementalIndices = theIncrementalIndices.size();
-
- if (numIncrementalIndices == 0)
- return;
-
- theInsertedDocsIndexDeltas.resize(numIncrementalIndices);
-
std::vector<XmlNode*>::const_iterator docIte = theInsertedDocs.begin();
std::vector<XmlNode*>::const_iterator docEnd = theInsertedDocs.end();
for (; docIte != docEnd; ++docIte)
{
- for (ulong i = 0; i < numIncrementalIndices; ++i)
+ for (csize i = 0; i < numIncrementalIndices; ++i)
{
store::IndexEntryCreator* docIndexer = theIndexEntryCreators[i].getp();
store::IndexDelta& indexDelta = theInsertedDocsIndexDeltas[i];
@@ -1939,22 +1958,6 @@
docIndexer->createIndexEntries((*docIte), indexDelta);
}
}
-
- theDeletedDocsIndexDeltas.resize(numIncrementalIndices);
-
- docIte = theDeletedDocs.begin();
- docEnd = theDeletedDocs.end();
-
- for (; docIte != docEnd; ++docIte)
- {
- for (ulong i = 0; i < numIncrementalIndices; ++i)
- {
- store::IndexEntryCreator* docIndexer = theIndexEntryCreators[i].getp();
- store::IndexDelta& indexDelta = theDeletedDocsIndexDeltas[i];
-
- docIndexer->createIndexEntries((*docIte), indexDelta);
- }
- }
}
@@ -1965,19 +1968,14 @@
********************************************************************************/
void CollectionPul::computeIndexDeltas(std::vector<store::IndexDelta>& deltas)
{
- ulong numIncrementalIndices = (ulong)theIncrementalIndices.size();
-
- if (numIncrementalIndices == 0)
- return;
-
- deltas.resize(numIncrementalIndices);
+ csize numIncrementalIndices = theIncrementalIndices.size();
std::set<XmlNode*>::const_iterator docIte = theModifiedDocs.begin();
std::set<XmlNode*>::const_iterator docEnd = theModifiedDocs.end();
for (; docIte != docEnd; ++docIte)
{
- for (ulong i = 0; i < numIncrementalIndices; ++i)
+ for (csize i = 0; i < numIncrementalIndices; ++i)
{
store::IndexEntryCreator* docIndexer = theIndexEntryCreators[i].getp();
store::IndexDelta& indexDelta = deltas[i];
@@ -1985,113 +1983,268 @@
docIndexer->createIndexEntries((*docIte), indexDelta);
}
}
-
- for (ulong i = 0; i < numIncrementalIndices; ++i)
- {
- store::IndexDelta& indexDelta = deltas[i];
-
- std::sort(indexDelta.begin(), indexDelta.end(), cmp);
- }
-}
-
-
-/*******************************************************************************
-
-********************************************************************************/
-void CollectionPul::refreshIndices()
-{
- csize numIncrementalIndices = theTruncatedIndices.size();
- for (csize idx = 0; idx < numIncrementalIndices; ++idx)
+}
+
+
+/*******************************************************************************
+
+********************************************************************************/
+void CollectionPul::cleanIndexDeltas()
+{
+ csize numIncrementalIndices = theIncrementalIndices.size();
+
+ for (csize idx = 0; idx < numIncrementalIndices; ++idx)
+ {
+ store::IndexDelta::iterator ite;
+ store::IndexDelta::iterator end;
+ store::IndexDelta* delta;
+ csize numApplied;
+
+ delta = &theInsertedDocsIndexDeltas[idx];
+ if (delta)
+ {
+ numApplied = theNumInsertedDocsIndexDeltasApplied[idx];
+ ite = delta->begin() + numApplied;
+ end = delta->end();
+ for (; ite != end; ++ite)
+ {
+ delete (*ite).second;
+ }
+ }
+
+ delta = &theAfterIndexDeltas[idx];
+ if (delta)
+ {
+ numApplied = theNumAfterIndexDeltasApplied[idx];
+ ite = delta->begin() + numApplied;
+ end = delta->end();
+ for (; ite != end; ++ite)
+ {
+ delete (*ite).second;
+ }
+ }
+
+ delta = &theDeletedDocsIndexDeltas[idx];
+ if (delta)
+ {
+ ite = delta->begin();
+ end = delta->end();
+ for (; ite != end; ++ite)
+ {
+ delete (*ite).second;
+ }
+ }
+
+ delta = &theBeforeIndexDeltas[idx];
+ if (delta)
+ {
+ ite = delta->begin();
+ end = delta->end();
+ for (; ite != end; ++ite)
+ {
+ delete (*ite).second;
+ }
+ }
+ }
+}
+
+
+/*******************************************************************************
+ Refresh the incrementally maintained indexes.
+********************************************************************************/
+void CollectionPul::refreshIndexes()
+{
+ csize numIncrementalIndices = theIncrementalIndices.size();
+
+ STORE_TRACE1("Refreshing indexes for collection "
+ << (theCollection ?
+ theCollection->getName()->getStringValue().c_str() :
+ "NULL"));
+
+ for (csize idx = 0; idx < numIncrementalIndices; ++idx)
+ {
+ ValueIndex* index = static_cast<ValueIndex*>(theIncrementalIndices[idx]);
+
+ STORE_TRACE2("Index size before do = "
+ << (!index->isTreeIndex() ? index->size() : 0));
+
+ store::IndexDelta& beforeDelta = theBeforeIndexDeltas[idx];
+ store::IndexDelta& afterDelta = theAfterIndexDeltas[idx];
+ store::IndexDelta& deletedDelta = theDeletedDocsIndexDeltas[idx];
+ store::IndexDelta& insertedDelta = theInsertedDocsIndexDeltas[idx];
+
+ csize& numBeforeApplied = theNumBeforeIndexDeltasApplied[idx];
+ csize& numAfterApplied = theNumAfterIndexDeltasApplied[idx];
+ csize& numDeletedApplied = theNumDeletedDocsIndexDeltasApplied[idx];
+ csize& numInsertedApplied = theNumInsertedDocsIndexDeltasApplied[idx];
+
+ store::IndexKey* key;
+ store::Item_t node;
+
+ store::IndexDelta::iterator ite;
+ store::IndexDelta::iterator end;
+
+ ite = beforeDelta.begin();
+ end = beforeDelta.end();
+ for (; ite != end; ++ite, ++numBeforeApplied)
+ {
+ index->remove((*ite).second, (*ite).first);
+ }
+
+ ite = afterDelta.begin();
+ end = afterDelta.end();
+ for (; ite != end; ++ite, ++numAfterApplied)
+ {
+ node = (*ite).first;
+ key = (*ite).second;
+
+ // If the index had its own key obj already, delete the key obj that was
+ // allocated during the delta creation.
+ if (index->insert((*ite).second, node))
+ {
+ assert(key != (*ite).second);
+ delete key;
+ }
+ }
+
+ STORE_TRACE2("deleted-delta size = " << deletedDelta.size());
+
+ ite = deletedDelta.begin();
+ end = deletedDelta.end();
+ for (; ite != end; ++ite, ++numDeletedApplied)
+ {
+ index->remove((*ite).second, (*ite).first);
+ }
+
+ STORE_TRACE2("inserted-delta size = " << insertedDelta.size());
+
+ ite = insertedDelta.begin();
+ end = insertedDelta.end();
+ for (; ite != end; ++ite, ++numInsertedApplied)
+ {
+ node = (*ite).first;
+ key = (*ite).second;
+
+ if (index->insert((*ite).second, node))
+ {
+ assert(key != (*ite).second);
+ delete key;
+ }
+ }
+
+ STORE_TRACE2("Index size after do = "
+ << (!index->isTreeIndex() ? index->size() : 0));
+ }
+
+ STORE_TRACE1("Refreshed indexes for collection "
+ << (theCollection ?
+ theCollection->getName()->getStringValue().c_str() :
+ "NULL")
+ << std::endl);
+}
+
+
+/*******************************************************************************
+
+********************************************************************************/
+void CollectionPul::undoRefreshIndexes()
+{
+ csize numIncrementalIndices = theIncrementalIndices.size();
+
+ STORE_TRACE1("Reverting indexes for collection "
+ << (theCollection ?
+ theCollection->getName()->getStringValue().c_str() :
+ "NULL")
+ << std::endl);
+
+ for (csize idx = 0; idx < numIncrementalIndices; ++idx)
+ {
+ ValueIndex* index = static_cast<ValueIndex*>(theIncrementalIndices[idx]);
+
+ STORE_TRACE2("Index size before undo = "
+ << (!index->isTreeIndex() ? index->size() : 0));
+
+ store::IndexDelta& beforeDelta = theBeforeIndexDeltas[idx];
+ store::IndexDelta& afterDelta = theAfterIndexDeltas[idx];
+ store::IndexDelta& insertedDelta = theInsertedDocsIndexDeltas[idx];
+ store::IndexDelta& deletedDelta = theDeletedDocsIndexDeltas[idx];
+
+ csize numBeforeApplied = theNumBeforeIndexDeltasApplied[idx];
+ csize numAfterApplied = theNumAfterIndexDeltasApplied[idx];
+ csize numDeletedApplied = theNumDeletedDocsIndexDeltasApplied[idx];
+ csize numInsertedApplied = theNumInsertedDocsIndexDeltasApplied[idx];
+
+ store::IndexDelta::reverse_iterator ite;
+ store::IndexDelta::reverse_iterator end;
+
+ ite = insertedDelta.rbegin() + (insertedDelta.size() - numInsertedApplied);
+ end = insertedDelta.rend();
+ for (; ite != end; ++ite)
+ {
+ index->remove((*ite).second, (*ite).first);
+ }
+
+ ite = deletedDelta.rbegin() + (deletedDelta.size() - numDeletedApplied);
+ end = deletedDelta.rend();
+ for (; ite != end; ++ite)
+ {
+ store::IndexKey* key = (*ite).second;
+
+ // If the index takes ownership of the key obj, set the key ptr to null
+ // so that the key obj will not be deleted during cleanIndexDeltas().
+ if (!index->insert(key, (*ite).first))
+ {
+ assert(key == (*ite).second);
+ (*ite).second = NULL;
+ }
+ }
+
+ ite = afterDelta.rbegin() + (afterDelta.size() - numAfterApplied);
+ end = afterDelta.rend();
+ for (; ite != end; ++ite)
+ {
+ index->remove((*ite).second, (*ite).first);
+ }
+
+ ite = beforeDelta.rbegin() + (beforeDelta.size() - numBeforeApplied);
+ end = beforeDelta.rend();
+ for (; ite != end; ++ite)
+ {
+ store::IndexKey* key = (*ite).second;
+
+ // If the index takes ownership of the key obj, set the key ptr to null
+ // so that the key obj will not be deleted during cleanIndexDeltas().
+ if (!index->insert(key, (*ite).first))
+ {
+ assert(key == (*ite).second);
+ (*ite).second = NULL;
+ }
+ }
+
+ STORE_TRACE2("Index size after undo = "
+ << (!index->isTreeIndex() ? index->size() : 0));
+ }
+
+ STORE_TRACE1("Reverted indexes for collection "
+ << (theCollection ?
+ theCollection->getName()->getStringValue().c_str() :
+ "NULL")
+ << std::endl);
+}
+
+
+/*******************************************************************************
+ The method is called from CollectionPul::finalizeUpdates()
+********************************************************************************/
+void CollectionPul::truncateIndexes()
+{
+ csize numTruncatedIndices = theTruncatedIndices.size();
+
+ for (csize idx = 0; idx < numTruncatedIndices; ++idx)
{
ValueIndex* index = static_cast<ValueIndex*>(theTruncatedIndices[idx]);
index->clear();
}
-
- numIncrementalIndices = theIncrementalIndices.size();
-
- for (csize idx = 0; idx < numIncrementalIndices; ++idx)
- {
- ValueIndex* index = static_cast<ValueIndex*>(theIncrementalIndices[idx]);
-
- //
- // Referesh the index w.r.t. modified docs.
- //
- ValueIndexCompareFunction keyCmp(index->getNumColumns(),
- index->getTimezone(),
- index->getCollations());
-
- store::IndexDelta& beforeDelta = theBeforeIndexDeltas[idx];
- store::IndexDelta& afterDelta = theAfterIndexDeltas[idx];
- store::IndexDelta& insertedDelta = theInsertedDocsIndexDeltas[idx];
- store::IndexDelta& deletedDelta = theDeletedDocsIndexDeltas[idx];
-
- store::IndexDelta::iterator beforeIte = beforeDelta.begin();
- store::IndexDelta::iterator beforeEnd = beforeDelta.end();
- store::IndexDelta::iterator afterIte = afterDelta.begin();
- store::IndexDelta::iterator afterEnd = afterDelta.end();
- store::IndexDelta::iterator insertedIte = insertedDelta.begin();
- store::IndexDelta::iterator insertedEnd = insertedDelta.end();
- store::IndexDelta::iterator deletedIte = deletedDelta.begin();
- store::IndexDelta::iterator deletedEnd = deletedDelta.end();
-
- while (beforeIte != beforeEnd && afterIte != afterEnd)
- {
- store::Item_t& beforeNode = (*beforeIte).first;
- store::Item_t& afterNode = (*afterIte).first;
- store::IndexKey* beforeKey = (*beforeIte).second;
- store::IndexKey*& afterKey = (*afterIte).second;
-
- if (beforeNode == afterNode)
- {
- if (!keyCmp.equal(beforeKey, afterKey))
- {
- index->remove(beforeKey, beforeNode);
- index->insert(afterKey, afterNode);
- }
-
- ++beforeIte;
- ++afterIte;
- }
- else if (beforeNode < afterNode)
- {
- index->remove(beforeKey, beforeNode);
- ++beforeIte;
- }
- else
- {
- index->insert(afterKey, afterNode);
- ++afterIte;
- }
- }
-
- while (beforeIte != beforeEnd)
- {
- index->remove((*beforeIte).second, (*beforeIte).first);
- ++beforeIte;
- }
-
- while (afterIte != afterEnd)
- {
- index->insert((*afterIte).second, (*afterIte).first);
- ++afterIte;
- }
-
- //
- // Referesh the index w.r.t. newly inserted docs.
- //
- for (; insertedIte != insertedEnd; ++insertedIte)
- {
- index->insert((*insertedIte).second, (*insertedIte).first);
- }
-
- //
- // Referesh the index w.r.t. deleted docs,
- //
- for (; deletedIte != deletedEnd; ++deletedIte)
- {
- index->remove((*deletedIte).second, (*deletedIte).first);
- }
- }
}
@@ -2100,6 +2253,37 @@
********************************************************************************/
void CollectionPul::applyUpdates()
{
+ csize numIncrementalIndices = theIncrementalIndices.size();
+
+#if 0
+ if (theCollection != NULL)
+ {
+ std::cout << "applying PUL for collection "
+ << theCollection->getName()->getStringValue() << std::endl;
+ }
+#endif
+
+ if (numIncrementalIndices > 0)
+ {
+ theBeforeIndexDeltas.resize(numIncrementalIndices);
+ theAfterIndexDeltas.resize(numIncrementalIndices);
+ theDeletedDocsIndexDeltas.resize(numIncrementalIndices);
+ theInsertedDocsIndexDeltas.resize(numIncrementalIndices);
+
+ theNumBeforeIndexDeltasApplied.resize(numIncrementalIndices);
+ theNumAfterIndexDeltasApplied.resize(numIncrementalIndices);
+ theNumInsertedDocsIndexDeltasApplied.resize(numIncrementalIndices);
+ theNumDeletedDocsIndexDeltasApplied.resize(numIncrementalIndices);
+
+ for (csize idx = 0; idx < numIncrementalIndices; ++idx)
+ {
+ theNumBeforeIndexDeltasApplied[idx] = 0;
+ theNumAfterIndexDeltasApplied[idx] = 0;
+ theNumInsertedDocsIndexDeltasApplied[idx] = 0;
+ theNumDeletedDocsIndexDeltasApplied[idx] = 0;
+ }
+ }
+
// Don't apply anything if the collection is going to be deleted.
if (!theDeleteCollectionList.empty())
return;
@@ -2214,6 +2398,71 @@
#endif
throw;
}
+
+#if 0
+ if (theCollection != NULL)
+ {
+ std::cout << "applied PUL for collection "
+ << theCollection->getName()->getStringValue() << std::endl << std::endl;
+ }
+#endif
+}
+
+
+/*******************************************************************************
+
+********************************************************************************/
+void CollectionPul::undoUpdates()
+{
+ if (!theIsApplied)
+ return;
+
+ try
+ {
+ undoList(theTruncateCollectionList);
+ undoList(theDeleteFromCollectionList);
+ undoList(theInsertIntoCollectionList);
+ undoList(theCreateCollectionList);
+
+#ifndef ZORBA_NO_XMLSCHEMA
+ // Undo validate-in-place validation
+ undoList(theRevalidateList);
+
+ // Undo apply-updates caused validation
+ if (theValidationPul)
+ {
+ undoList(static_cast<PULImpl *>(theValidationPul.getp())->theValidationList);
+ }
+#endif
+
+ // Undo text node merging
+ std::vector<TextNodeMerge>::reverse_iterator rit = theMergeList.rbegin();
+ std::vector<TextNodeMerge>::reverse_iterator rend = theMergeList.rend();
+ for (; rit != rend; ++rit)
+ {
+ TextNodeMerge merge = (*rit);
+ XmlNode* newTextNode = merge.theParent->getChild(merge.thePos);
+ ZORBA_ASSERT(newTextNode->getNodeKind()== store::StoreConsts::textNode);
+
+ newTextNode->detach();
+
+ for (csize j = 0; j < merge.theMergedNodes.size(); ++j)
+ merge.theMergedNodes[j]->connect(merge.theParent, merge.thePos + j);
+ }
+ theMergeList.clear();
+
+ undoList(theDeleteList);
+ undoList(theReplaceContentList);
+ undoList(theReplaceNodeList);
+ undoList(theInsertList);
+ undoList(theDoFirstList);
+
+ undoRefreshIndexes();
+ }
+ catch (...)
+ {
+ ZORBA_FATAL(0, "Unexpected error during pul undo");
+ }
}
@@ -2228,9 +2477,7 @@
{
try
{
- // Refresh each incrementally maintained index using its before and after
- // deltas.
- refreshIndices();
+
Follow ups
