zorba-coders team mailing list archive
-
zorba-coders team
-
Mailing list archive
-
Message #26783
[Merge] lp:~paul-lucas/zorba/pjl-misc into lp:zorba
Paul J. Lucas has proposed merging lp:~paul-lucas/zorba/pjl-misc into lp:zorba.
Commit message:
Discovered a bug where strings like street addressed, e.g., "870 Market Street", will cause an exception to be thrown due to the mis-guessing about what kind of JSON token a string really is. Fixed.
Requested reviews:
Paul J. Lucas (paul-lucas)
For more details, see:
https://code.launchpad.net/~paul-lucas/zorba/pjl-misc/+merge/208907
Discovered a bug where strings like street addressed, e.g., "870 Market Street", will cause an exception to be thrown due to the mis-guessing about what kind of JSON token a string really is. Fixed.
--
https://code.launchpad.net/~paul-lucas/zorba/pjl-misc/+merge/208907
Your team Zorba Coders is subscribed to branch lp:zorba.
=== modified file 'src/runtime/csv/csv_impl.cpp'
--- src/runtime/csv/csv_impl.cpp 2014-02-28 01:46:12 +0000
+++ src/runtime/csv/csv_impl.cpp 2014-03-01 01:55:17 +0000
@@ -162,8 +162,31 @@
istringstream iss;
iss.ios::rdbuf( &buf );
json::lexer lex( iss );
- return lex.next( ptoken, false ) ?
- json::map_type( ptoken->get_type() ) : json::none;
+ if ( !lex.next( ptoken, false ) )
+ return json::none;
+ json::token::type const tt = ptoken->get_type();
+ if ( tt == json::token::number ) {
+ //
+ // The JSON lexer will stop lex'ing as soon as it finds a valid token. For
+ // strings that start with numbers, e.g., "870 Market St", the lexer will
+ // return "870" as a number token but we need to know the type of the whole
+ // string 's'; hence we need to do an extra check for numer tokens.
+ //
+ // If the length(t) < length(s), remove trailing whitespace from 's' and
+ // check again. If length(t) < length(s2), it means that that the token is
+ // really a string; if length(t) == length(s2), then the length difference
+ // was caused only by whitespace that can be ignored and the token is still
+ // a number.
+ //
+ json::token::value_type const &value = ptoken->get_value();
+ if ( value.size() < s.size() ) {
+ zstring s2;
+ ascii::trim_end_space( s, &s2 );
+ if ( value.size() < s2.size() )
+ return json::string;
+ }
+ }
+ return json::map_type( tt );
}
static void set_keys( store::Item_t const &item, vector<store::Item_t> *keys,
=== added file 'test/rbkt/ExpQueryResults/zorba/csv/csv-parse-address.xml.res'
--- test/rbkt/ExpQueryResults/zorba/csv/csv-parse-address.xml.res 1970-01-01 00:00:00 +0000
+++ test/rbkt/ExpQueryResults/zorba/csv/csv-parse-address.xml.res 2014-03-01 01:55:17 +0000
@@ -0,0 +1,3 @@
+{
+ "Street" : "870 Market Street"
+}
=== added file 'test/rbkt/Queries/zorba/csv/csv-parse-address.jq'
--- test/rbkt/Queries/zorba/csv/csv-parse-address.jq 1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/csv/csv-parse-address.jq 2014-03-01 01:55:17 +0000
@@ -0,0 +1,7 @@
+import module namespace csv = "http://zorba.io/modules/json-csv";
+
+let $csv := "870 Market Street"
+let $options := { "field-names" : [ "Street" ] }
+return csv:parse( $csv, $options )
+
+(: vim:set syntax=xquery et sw=2 ts=2 :)
=== added file 'test/rbkt/Queries/zorba/csv/csv-parse-address.spec'
--- test/rbkt/Queries/zorba/csv/csv-parse-address.spec 1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/csv/csv-parse-address.spec 2014-03-01 01:55:17 +0000
@@ -0,0 +1,1 @@
+Serialization: indent=yes
References