opensubscriber
   Find in this group all groups
 
Unknown more information…

a : advanced-dotnet@discuss.develop.com 28 August 2008 • 8:03AM -0400

Re: [ADVANCED-DOTNET] Converting doubles into integers without rounding errors
by David Nicholson

REPLY TO AUTHOR
 
REPLY TO GROUP




Since there seems to be no clear solution so far, I've gone back to the
beginning. I suggest you get more information by writing some code to
display your numbers to higher precision. My expectation is that the
137.89 is actually 137.889999 (probably more nine's). So your code below
converts this to 13788.9999, then truncates to 137.88.

So add a small value (e.g. .0001) first. Then the outcomes for numbers
above and below the intended number look something like:

137.889999 -> 137.890099 -> 13789.0099 -> 13789
137.890000 -> 137.890100 -> 13789.0100 -> 13789
137.890001 -> 137.890101 -> 13789.0101 -> 13789

David.


On Mon, 25 Aug 2008 12:27:52 -0400, Eddie Lascu <elascu@IBIG...>
wrote:

>Hello everyone,
>
>I have some objects that contain an amount field that is declared as
double.
>Since it contains amounts, it always has only two decimal digits that are
>significant. During the process I need to convert that double into an
>integer by removing the decimal point. For example, $78.59 should be
>converted to integer 7859 and $101.53 to 10153. in my code I have
>uint nIntAmount = (uint)(objMyObject.Amount * 100);
>
>The problem I am facing is that sometimes, very rarely, there is a
rounding
>error that is introduced and the integer obtained is off by a cent (plus
or
>minus). For example, this is a line that was traced in my log file:
>
>"Updating the batch with $137.89 as the amount in the transaction. This
>amount was converted to 13788."
>
>Can either of you suggest a different way to convert the amounts in
integers
>without this nagging rounding error?
>
>Any help will be appreciated,
>
>Eddie
>
>===================================
>This list is hosted by DevelopMentorĀ®  http://www.develop.com
>
>View archives and manage your subscription(s) at
http://discuss.develop.com

===================================
This list is hosted by DevelopMentorĀ®  http://www.develop.com

View archives and manage your subscription(s) at http://discuss.develop.com

Bookmark with:

Delicious   Digg   reddit   Facebook   StumbleUpon

Related Messages

opensubscriber is not affiliated with the authors of this message nor responsible for its content.