Incorrect dumping of parsed numbers with exponents, but without decimal places

[nlohmann]

The new floating point representation (#201) has a bug when it comes to negative exponents. Number 2342e-2 is correctly duped as 23.42, but when it is parsed from a string, the output is 2e01.

Example code:

json j2a = 2342e-2;
json j2b = json::parse("2342e-2");

CHECK(j2a.dump() == "23.42");
CHECK(j2b.dump() == "2342e-2");

Output:

FAILED:
  CHECK( j2b.dump() == "2342e-2" )
with expansion:
  "2e01" == "2342e-2"

@twelsby Any idea about this?

[nlohmann]

There is also an issue with other parsed numbers with exponents, but without decimal separator:

2342e2 and 23420e1 both serialize to 2e05.

[nlohmann]

❗ This bug is currently a blocker for the 2.0.0 release, because the serialization of parsed numbers is not correct.

[nlohmann]

@twelsby @gregmarr - Any idea how to fix this?

[camerondruyor]

Caveat: I'm new to this project (like 1 hour), so please forgive my ignorance.

I drilled down into dump() to try to isolate the problem to either the parser or the formatting. For this problem case, the underlying data is:

m_type.bits.exp_cap    = 0 
m_type.bits.exp_plus   = 0 
m_type.bits.has_exp    = 1 
m_type.bits.parsed     = 1 
m_type.bits.precision  = 0 
m_type.bits.type       = 7 
m_value                = 23.420000

It seems to me, that the precision should be 4, not 0. Am I understanding this correctly?

[camerondruyor]

Never mind, after looking through the lexer, I realized precision isn't sig figs, but number-of-figs-after-radix in this context. In which case, 2342e-2 is being lexed and parsed correctly, so I'll look back into the dump formatting again.

[camerondruyor]

Since we're using snprintf with "%.*e" to generate the string, wouldn't we want 2342e-2 to dump as 2.342e01? (since it is going to print one leading digit for scientific notation)

If I detect this particular case, and pack the buffer with precision=3, I can get the test to pass.

My spidey sense tells me that we might need a little more logic in the lexer at this point:

            // If no radix point was found then precision would now be set to
            // the number of digits, which is wrong - clear it.
            result.m_type.bits.precision = precision & found_radix_point;

This logic works fine for test cases like 10e3. They actually only have 1 significant figure anyway, so the precision can be set to 0.

[camerondruyor]

Ok, I have a working fix.

It passes all the old tests, and now we get 2.342e01 instead of 2e01.

I need to clean up my patch before submitting it, but it's getting late (1:30am here), so I'll do that tomorrow.

[nlohmann]

Thanks for looking at the issue @KAMTRON! I am looking forward to the fix. Good night! :)

[camerondruyor]

I have a dumb question @nlohmann .

How do I

make re2c

in order to generate json.hpp? I don't see that target in the makefile anywhere. I have built re2c and run it on json.hpp.re2c, but that gives me the lexer with all the gotos.

I feel like I'm missing something really simple.

[nlohmann]

@KAMTRON,

running make re2c should execute

re2c --bit-vectors --nested-ifs --no-debug-info src/json,hpp.re2c | gsed '1d' > src/json.hpp

The Makefile has src/json.hpp.re2c as prerequisite, so maybe it is not building because that file was not changed.

If it still does not work, please let me know.

[camerondruyor]

@nlohmann Thanks. I never did get it to work with make, but I could get by running the command manually.