Floting point problem


Author
Message
mrg4
mrg4
Distinguished Member (3.2K reputation)Distinguished Member (3.2K reputation)Distinguished Member (3.2K reputation)Distinguished Member (3.2K reputation)Distinguished Member (3.2K reputation)Distinguished Member (3.2K reputation)Distinguished Member (3.2K reputation)Distinguished Member (3.2K reputation)Distinguished Member (3.2K reputation)
Group: Forum Members
Posts: 19, Visits: 218
Hello,

we have a task where the participant earns points according to which key he presses.
In one  case the participant receives 1 point, in the other case he gets 0.1 points. After each trial a feedback is given, saying:

"You received 0.1 / 1 points. Your total score is: XY.XY"

We therefore defined one value element for the gain in the recent trial and one counting the total score.
In case the participant receives 1 point, everything works perfecty fine, but due to the fact that the computer works with floating point, after the first time 0.1 points are added to the score, our feedback looks like e.g.:

"You received 0.10000000000000001 points. Your total score is: 5.10000000000000001"

We already tried to solve the problem by using rounding, but none of our attempts worked. We now use 0.5 / 5 points instead of 0.1 / 1, and it works perfectly.

This is thought to be a temporary solution, so we're hoping anyone is able to help us.


Thank you very much in advance.

Best regards





Dave
Dave
Supreme Being (781K reputation)Supreme Being (781K reputation)Supreme Being (781K reputation)Supreme Being (781K reputation)Supreme Being (781K reputation)Supreme Being (781K reputation)Supreme Being (781K reputation)Supreme Being (781K reputation)Supreme Being (781K reputation)
Group: Administrators
Posts: 10K, Visits: 50K
This is a completely natural consequence of how computers represent real numbers (cf. http://www.millisecond.com/forums/FindPost7984.aspx ). Your best bet is to use the format() function to limit the output of digits after the dot.

-> format("%.2f", values.myvalue)
GO

Merge Selected

Merge into selected topic...



Merge into merge target...



Merge into a specific topic ID...




Reading This Topic

Explore
Messages
Mentions
Search