c2c-oerpscenario team mailing list archive

[Cloves Almeida <cjalmeida@xxxxxxxxx>]

The algorithm in #15 fails in negative numbers and when rouding higher
precision numbers. An adjusted version:

the "cmp(f,0)*pow(2,-52)" is the signal adjusted machine epsilon [1].
Contrary to Wikipedia, python's epsilon is 2^-52 instead of 2^53

def roundf(f, prec=0):
    return round(f + cmp(f,0)*pow(2,-52),prec)


>>> roundf(0.125,2)
0.13
>>> roundf(-0.125,2)
-0.13
>>> roundf(-0.124999999,2)
-0.12
>>> roundf(1.515,2)
1.52
>>> roundf(0.575,2)
0.57999999999999996
>>> str(roundf(0.575,2))
'0.58'
>>> roundf(2.675,2)
2.6800000000000002
>>> str(roundf(2.675,2))
'2.68'
>>> str(roundf(-2.675,2))
'-2.68'

[1] http://en.wikipedia.org/wiki/Machine_epsilon

-- 
You received this bug notification because you are a member of C2C
OERPScenario, which is subscribed to OpenERP Project Group.
https://bugs.launchpad.net/bugs/882036

Title:
  rounding error

Status in OpenERP Server:
  New

Bug description:
  Concerns 6.0 and trunk.
  If you define a precision of 0.01, the rounding of 0.125 must be 0.13 and not 0.12. The error is in the call of the format string "%.2f"%val which introduces a mathematical error. The round function must be called to apply the correct rounding before formatting the string. It should be:
  "%.2f"%round(val*100)/100
  Fix class digits_change of class float in the server. BUT fix also the gtk client AND web client as they all have that error (I let you find the right line)

To manage notifications about this bug go to:
https://bugs.launchpad.net/openobject-server/+bug/882036/+subscriptions