test floating point casting better#1265
Conversation
| assert_eq!((u32::MAX-127) as f32 as u32, u32::MAX); // rounding loss | ||
| assert_eq!((u32::MAX-128) as f32 as u32, u32::MAX-255); // rounding loss |
There was a problem hiding this comment.
@hanna-kruppe do these two (and the two "rounding loss" tests below) look like correct behavior to you? I experimentally determined these to be the closest to u32::MAX where the output of the roundtrip cast changes.
There was a problem hiding this comment.
@jodysankey you also seem to know floating point encoding stuff quite well, would be nice if you could have a look.
There was a problem hiding this comment.
Yes, they look exactly right to me. These are transition points from "odd significant + slightly less than 0.5 ULP" (which gets rounded down to uint::MAX - 1 ULP) to "odd significant + 0.5 ULP" (which is rounded up, so saturation kicks in when casting back to integer).
There was a problem hiding this comment.
@hanna-kruppe I only understood half of that, but that half sounds good. ;) Thanks!
|
@bors r+ |
|
📌 Commit b8a817f has been approved by |
|
☀️ Test successful - checks-travis, status-appveyor |
3 participants
No description provided.