Stream of objects on a HUGE homogeneous array of objects (or on a map with tons of keys to homogeneous objects)

[lonevetad]

I have a huge JSON file containing an array with a lot of objects, like 1000, all with the same structure (they derive from the same class).
Rather than parsing the whole array first and occupy a lot of memory, I'd like to iterate through it and parse one object at a time. Similarly, I could have a giant map with tons of keys, all mapping to similarly-structured objects.
Can a new function be defined to emit a stream / iterator to one-by-one parsed objects?

[gistrec]

nlohmann/json's parser callback is built for this. Returning false from the callback discards a parsed value and frees its memory before the next element is built.

For a top-level array of objects, object_end at depth == 1 fires with each completed object in parsed:

#include <fstream>
#include <nlohmann/json.hpp>

std::ifstream file("big.json");
nlohmann::json::parse(
    file,
    [&](int depth, nlohmann::json::parse_event_t event, nlohmann::json& parsed) {
        if (depth == 1 && event == nlohmann::json::parse_event_t::object_end) {
            process(parsed);  // handle one element
            return false;     // discard before next element is retained
        }
        return true;
    }
);

The file is read as a stream, not buffered. Each object still needs to fit in memory while being assembled — this pattern is the right fit when individual records fit but the whole array does not.

The same depth == 1 && object_end pattern works for a top-level map of objects (each value-object also ends at depth 1).

For truly giant individual records, see sax_parse.