Null bytes in files are treated like EOF

[nlohmann]

• What is the issue you have?

When reading parsing JSON from a file, null bytes are treated as EOF. The reason for this is explicit code in the JSON lexer to treat \0 as EOF to properly parse string literals.

• Please describe the steps to reproduce the issue. Can you provide a small but working code example?

Parse file https://github.com/nst/JSONTestSuite/blob/master/test_parsing/n_multidigit_number_then_00.json, e.g. by

std::ifstream f("n_multidigit_number_then_00.json");
json::parse(f);

• What is the expected behavior?

Parse error: unexpected null byte after reading 123.

• And what is the actual behavior instead?

The file is successfully parsed as number 123.

• Which compiler and operating system are you using? Is it a supported compiler?

Apple LLVM version 9.1.0.

• Did you use a released version of the library or the version from the develop branch?

develop.

• If you experience a compilation error: can you compile and run the unit tests?

N/A

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[FrancoisChabot]

I have a working fix for this, however, it causes a knock-on that I believe is desirable.

The following test(and a few other almost identical ones) now fail:

    SECTION("from std::array")
    {
        std::array<uint8_t, 5> v { {'t', 'r', 'u', 'e'} };
        json j;
        json::parser(nlohmann::detail::input_adapter(std::begin(v), std::end(v))).parse(true, j);
        CHECK(j == json(true));
    }

This makes sense as the explicit 0 at the end of the array is now considered to be part of the parsed string.

I would just change the test to be a std::array<uint8_t, 4> instead.

Is that fine?

Edit: I've attached my work-in-progress as a PR for reference.

[nlohmann]

Thanks for the PR! I added some comments, and I need to think whether there is another use case that needs to be tested.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[nlohmann]

@FrancoisChabot It would be great if you could have another look at #1452.