banking-addons-team team mailing list archive

Thread
Date

[Bug 1285075] [NEW] bank-statement-reconcile: import of bank statement fails to sniff the csv

To: banking-addons-team@xxxxxxxxxxxxxxxxxxx
From: "Guewen Baconnier @ Camptocamp" <1285075@xxxxxxxxxxxxxxxxxx>
Date: Wed, 26 Feb 2014 10:22:01 -0000
Reply-to: Bug 1285075 <1285075@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx

Public bug reported:

When importing a CSV file with account_statement_base_import, it tries to auto-detect the dialect of the CSV file to import.
I have some CSV files that cannot be detected, ending with an error: Error: Could not determine delimiter

Example of CSV that can't  be sniffed (with many more lines):

transaction_id,date,amount,commission_amount,label
1524624,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524635,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524637,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524643,2014-02-18 00:00:00,276.000000000,4.14,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524660,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524662,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524693,2014-02-18 00:00:00,187.000000000,2.81,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524704,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524712,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524730,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524732,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524735,2014-02-18 00:00:00,187.000000000,2.81,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524737,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524740,2014-02-18 00:00:00,187.000000000,2.81,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524751,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524762,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524766,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524773,2014-02-18 00:00:00,187.000000000,2.81,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524775,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889

Currently the code is:

    sniffer = csv.Sniffer() 
    pos = utf8_data.tell()
    sample_data = utf8_data.read(1024)
    utf8_data.seek(pos) dialect = sniffer.sniff(sample_data, delimiters=',;\t')

>From my analysis, it seems that the sample of 1024 bytes is insufficient
to guess the delimiter for this type of files. Raising it to 2048 bytes
allowed it to guess the delimiter correctly.

** Affects: banking-addons
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Banking
Addons Core Editors, which is subscribed to Banking Addons.
https://bugs.launchpad.net/bugs/1285075

Title:
  bank-statement-reconcile: import of bank statement fails to sniff the
  csv

Status in OpenERP Banking Addons:
  New

Bug description:
  When importing a CSV file with account_statement_base_import, it tries to auto-detect the dialect of the CSV file to import.
  I have some CSV files that cannot be detected, ending with an error: Error: Could not determine delimiter

  Example of CSV that can't  be sniffed (with many more lines):

  transaction_id,date,amount,commission_amount,label
  1524624,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524635,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524637,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524643,2014-02-18 00:00:00,276.000000000,4.14,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524660,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524662,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524693,2014-02-18 00:00:00,187.000000000,2.81,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524704,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524712,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524730,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524732,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524735,2014-02-18 00:00:00,187.000000000,2.81,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524737,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524740,2014-02-18 00:00:00,187.000000000,2.81,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524751,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524762,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524766,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524773,2014-02-18 00:00:00,187.000000000,2.81,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
  1524775,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889

  Currently the code is:

      sniffer = csv.Sniffer() 
      pos = utf8_data.tell()
      sample_data = utf8_data.read(1024)
      utf8_data.seek(pos) dialect = sniffer.sniff(sample_data, delimiters=',;\t')

  From my analysis, it seems that the sample of 1024 bytes is
  insufficient to guess the delimiter for this type of files. Raising it
  to 2048 bytes allowed it to guess the delimiter correctly.

To manage notifications about this bug go to:
https://bugs.launchpad.net/banking-addons/+bug/1285075/+subscriptions

Follow ups

[Bug 1285075] Re: bank-statement-reconcile: import of bank statement fails to sniff the csv
From: Yannick Vaucher @ Camptocamp, 2014-02-26
[Bug 1285075] [NEW] bank-statement-reconcile: import of bank statement fails to sniff the csv
From: Guewen Baconnier @ Camptocamp, 2014-02-26

References

[Bug 1285075] [NEW] bank-statement-reconcile: import of bank statement fails to sniff the csv
From: Guewen Baconnier @ Camptocamp, 2014-02-26