Support for instantiating Half-Typed-grid from stored float or double grids

[konivo]

The main motivation for this patch is to be able to load half-typed grids directly from files containing non-half-typed grids in order to keep the process peak memory low. When having large grids or large number of grids this can be important.

The PR does (atm) the conversion according to the following rules:

stored grid value type	desired conversion	returned grid
float	half	half
double	half	half
vec2f	half	vec2h
vec3f	half	vec3h
vec2d	half	vec2h
vec3d	half	vec3h

The above rules can be extended if desirable, and in the future there could be also 'store' counterpart to do conversion into other grid types while storing. That could potentially replace the need for "saveFloatAsHalf" flag.

The core implementation has been already tested with various vdb files, some of them using saveFloatAsHalf, and the approach seems to be working without issues.

The implementation is straightforward and follows these steps:

• User specifies desired conversion when calling File::open or when instantiating Stream
• Desired conversion information then propagates into StreamMetadata
• GridDescriptor::read picks up the conversion from the metadata and from io::ConvertingReaderFactory it gets a reader and the resulting grid valuetype. It then instantiates a grid with this new valuetype instead of using the original value type. If there is no suitable reader for the desired conversion, the method instantiates a grid with the original valuetype.
• The reader returned from io::ConvertingReaderFactory is later also stored in StreamMetadata and used during topology/buffers loading.

[linux-foundation-easycla]

• ❌ - login: @konivo / name: Ivo Kondapaneni . The commit (b1f1578) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

[apradhana]

Hi, @konivo , I was trying to test this PR on my end using this function:

void testDirectConversionToHalfGrid() {
    using namespace openvdb;
    std::string fileName = "./dragon.vdb";

    io::File file(fileName);
    file.open(false /* delay load*/, io::MappedFile::Notifier(), io::Archive::ScalarConversion::Half);

    HalfGrid::registerGrid();

    HalfGrid::Ptr grid;
    // Loop over the names of all of the grids in the file.
    for (io::File::NameIterator nameIter = file.beginName();
        nameIter != file.endName(); ++nameIter)
    {
        std::string gridName = nameIter.gridName();
        grid = gridPtrCast<HalfGrid>(file.readGrid(gridName));
        std::cout << "gridName = " << gridName << "grid = " << grid << "\n";
    }
}

My expectation is that grid should be non-nullptr. However, I couldn't make this to work. Am I missing something here? Also, with the release of VDB-13, HalfGrid registration should already happen in openvdb::initialize(), so if you rebase on top of the master branch, HalfGrid::registerGrid() shouldn't be necessary.

@danrbailey and @Idclip for viz.

[apradhana]

The test I proposed for converting float to half passes for commit ID b1f157. However, there are 12 unit-tests that are not-passing, which can be due to ABI change introduced in this commit (i.e. adding new member variables in StreamMetadata::Impl). The uni-tests that are not passing are TestLeafIOTest.testBufferInt:TestLeafIOTest.testBufferFloat:TestLeafIOTest.testBufferDouble:TestLeafIOTest.testBufferByte:TestLeafIOTest.testBufferVec3R:TestFile.testWriteGrid:TestFile.testWriteMultipleGrids:TestFile.testReadGridDescriptors:TestFile.testEmptyGridIO:TestFile.testDelayedLoadMetadata:TestGridDescriptor.testIO:TestTree.testHalf:TestTree.testIO.

@konivo and I decided the following action item: create convertingReader in GridDescriptor.cc while we are creating grid, then pass it down to Archive::doReadGrid and later to RootNode::readTopology and LeafNode::read. @apradhana is to test this approach and present it in the next TSC meeting.

[apradhana]

Modifying StreamMetadata::Impl changes the ABI and introduces segfaults when performing I/O. See comment at the beginning.

[apradhana]

What makes more sense is to move this to readTopologyWithValueType since this function is templated with SourceValueT. In fact, I suspect that for readTopologyWithValueType, we should be able to make the conversion from Real -> Half type to work without ConvertingReader by providing the correct type in SourceValueT.

[danrbailey]

The test I proposed for converting float to half passes for commit ID b1f157. However, there are 12 unit-tests that are not-passing, which can be due to ABI change introduced in this commit (i.e. adding new member variables in StreamMetadata::Impl). The uni-tests that are not passing are TestLeafIOTest.testBufferInt:TestLeafIOTest.testBufferFloat:TestLeafIOTest.testBufferDouble:TestLeafIOTest.testBufferByte:TestLeafIOTest.testBufferVec3R:TestFile.testWriteGrid:TestFile.testWriteMultipleGrids:TestFile.testReadGridDescriptors:TestFile.testEmptyGridIO:TestFile.testDelayedLoadMetadata:TestGridDescriptor.testIO:TestTree.testHalf:TestTree.testIO.

@konivo and I decided the following action item: create convertingReader in GridDescriptor.cc while we are creating grid, then pass it down to Archive::doReadGrid and later to RootNode::readTopology and LeafNode::read. @apradhana is to test this approach and present it in the next TSC meeting.

I think this might be a good time for me to present my progress on re-architecturing VDB I/O and discuss some possible future directions. Under this new proposal, some of what is being discussed / implemented here will remain the same, but there is also a lot that would need to change. At this point, I would prefer that we include this as new functionality in the re-design as opposed to extending the existing VDB I/O.

[apradhana]

That sounds great, @danrbailey !