banking-addons-team team mailing list archive

["Alexandre Fayolle @ camptocamp" <alexandre.fayolle@xxxxxxxxxxxxxx>]

Alexandre Fayolle @ camptocamp has proposed merging lp:~banking-addons-team/banking-addons/bank-statement-reconcile-fix-1082402 into lp:banking-addons/bank-statement-reconcile-61.

Requested reviews:
  Banking Addons Team (banking-addons-team)
Related bugs:
  Bug #1082402 in Banking Addons: "account_statement_base_import: crash when a CSV file with MacOS style EOL is uploaded"
  https://bugs.launchpad.net/banking-addons/+bug/1082402
  Bug #1082412 in Banking Addons: "account_statement_base_import: CSV separator is hard coded to ';'"
  https://bugs.launchpad.net/banking-addons/+bug/1082412

For more details, see:
https://code.launchpad.net/~banking-addons-team/banking-addons/bank-statement-reconcile-fix-1082402/+merge/135943

Fix two bugs related to CSV file processing :

* macOS line termination handling
* auto detection of the delimiter
-- 
https://code.launchpad.net/~banking-addons-team/banking-addons/bank-statement-reconcile-fix-1082402/+merge/135943
Your team Banking Addons Team is requested to review the proposed merge of lp:~banking-addons-team/banking-addons/bank-statement-reconcile-fix-1082402 into lp:banking-addons/bank-statement-reconcile-61.

=== modified file 'account_statement_base_import/parser/file_parser.py'
--- account_statement_base_import/parser/file_parser.py	2012-09-26 12:51:06 +0000
+++ account_statement_base_import/parser/file_parser.py	2012-11-23 16:43:21 +0000
@@ -19,8 +19,6 @@
 ##############################################################################
 
 from openerp.tools.translate import _
-import base64
-import csv
 import tempfile
 import datetime
 from parser import BankStatementImportParser
@@ -108,20 +106,16 @@
         return True
 
 
-    def _parse_csv(self, delimiter=';'):
+    def _parse_csv(self):
         """
-        :return: dict of dict from csv file (line/rows)
+        :return: list of dict from csv file (line/rows)
         """
         csv_file = tempfile.NamedTemporaryFile()
         csv_file.write(self.filebuffer)
-        # We ensure that cursor is at beginig of file
-        csv_file.seek(0)
-        reader = UnicodeDictReader(
-                    open(csv_file.name).readlines(),
-                    delimiter=delimiter,
-                    fieldnames=self.fieldnames
-        )
-        return [x for x in reader]
+        csv_file.flush()
+        with open(csv_file.name, 'rU') as fobj: 
+            reader = UnicodeDictReader(fobj, fieldnames=self.fieldnames)
+            return list(reader)
 
     def _parse_xls(self):
         """

=== modified file 'account_statement_base_import/parser/parser.py'
--- account_statement_base_import/parser/parser.py	2012-06-20 14:10:01 +0000
+++ account_statement_base_import/parser/parser.py	2012-11-23 16:43:21 +0000
@@ -22,7 +22,12 @@
 import csv
 
 def UnicodeDictReader(utf8_data, **kwargs):
-    csv_reader = csv.DictReader(utf8_data, **kwargs)
+    sniffer = csv.Sniffer()
+    pos = utf8_data.tell()
+    sample_data = utf8_data.read(1024)
+    utf8_data.seek(pos)
+    dialect = sniffer.sniff(sample_data, delimiters=',;\t')
+    csv_reader = csv.DictReader(utf8_data, dialect=dialect, **kwargs)
     for row in csv_reader:
         yield dict([(key, unicode(value, 'utf-8')) for key, value in row.iteritems()])
 
@@ -210,4 +215,4 @@
         if cls.parser_for(parser_name):
             return cls(parser_name, *args, **kwargs)
     raise ValueError
-            
\ No newline at end of file
+