Views are not working (populating, or triggering) properly with $unionWith

[franky-continu]

Versions

• NodeJS: v18.16.0
• mongodb-memory-server-*: 11.0.1
• mongodb(the binary version): 0.0.0 <!--Let this stay at "0", when assuming default, also see
• mongodb(the js package): 0.0.0
• mongoose: 8.16.2
• system: MacOS

package: mongo-memory-server

What is the Problem?

Querying View (for all content) returns 0 documents, while individual content collections have content.

Code Example

async function bootstrapContentView() {
  const viewName = 'contents';
  const sourceCollection = 'articles';
  const viewPipeline = [
    {
      $addFields: {
        recompletable: true
      }
    },
    {
      $unionWith: {
        coll: "media",
        pipeline: [
          {
            $addFields: {
              recompletable: true
            }
          }
        ]
      }
    },
    {
      $unionWith: {
        coll: "importedcontents",
        pipeline: [
          {
            $addFields: {
              recompletable: false
            }
          }
        ]
      }
    }
  ];
  const db = mongoose.connection.db;
  const existing = await db.listCollections({ name: viewName }, { nameOnly: true }).toArray();
  if (existing.length > 0) {
    try {
      log.info(`View ${ viewName } already exists.`);
      const updateResult = await db.command({ collMod: viewName, viewOn: sourceCollection, pipeline: viewPipeline });
      log.info(`View ${ viewName } updated with result: ${ JSON.stringify(updateResult) }`);
      return;
    } catch (error) {
      // Handle other potential errors
      log.error(error, `Error bootstrapping updating view: ${ error.message }`);
    }
  }
  try {
    const createResult = await db.command({ create: viewName, viewOn: sourceCollection, pipeline: viewPipeline });
    log.info(`View ${ viewName } created with result: ${ JSON.stringify(createResult) }`);
    return;
  } catch (error) {
    if (error.codeName === 'NamespaceExists' || error.code === 48 || error.code === 59) {
      log.info(`View ${ viewName } already exists. `);
    } else {
      // Handle other potential errors
      log.error(error, `Error bootstrapping view: ${ error.message }`);
    }
  }
}

export async function connectToDB() {
  // followed some recommendations from:
  //  -- https://gist.github.com/mongolab-org/9959376
  //  -- http://mongoosejs.com/docs/connections.html
  //  -- http://mongodb.github.io/node-mongodb-native/2.1/api/Server.html
  //  -- http://mongodb.github.io/node-mongodb-native/2.1/api/ReplSet.html

  const options = {
    maxPoolSize: config.db.maxConnectionPoolSize, // per container connection limit. We currently have around a 500 total connection limit on the mongo server side
    connectTimeoutMS: 30000,
    family: 4
  };

  if (config.testing && config.containerized === false) {
    if (!mongoMemoryDb) {
      log.info('Configuring Mongo Memory Server');

      /* eslint-disable-next-line global-require */
      const { MongoMemoryReplSet } = require('mongodb-memory-server');
      mongoMemoryDb = await MongoMemoryReplSet.create({
        binary: { downloadDir: 'node_modules/.cache/mongodb-memory-server/mongodb-binaries' },
        count: 1,
      });
      // only override url if we're not running tests inside a docker container. If we are, tests
      // will connect to a mongo container instead of mongo memory server
      config.db.connectionUrl = mongoMemoryDb.getUri('TestingDB');
    }
  }

  try {
    const connection = await mongoose.connect(config.db.connectionUrl, options);
    log.info(`mongoose.connect Called Successfully. ${ config.db.connectionUrl }`);
    await bootstrapContentView();
    return connection;
  } catch (err) {
    log.error(err, 'Error connecting to MongoDB');
    throw err;
  }
}
// Filtered query returns 0 documents
//   const contents = await AllContent.find({_id: {$in: contentIds}}, '_id type recompletable').lean().exec();
//   console.log('contents to check for recompletion', contents);
//   $ contents to check for recompletion []

// Unfiltered query returns 0 documents
//   const cont = await AllContent.find({}, '_id type recompletable').limit().lean().exec();
//   console.log('all content no filter', cont);
//   $ all content no filter []

NOT in the code above but if you do the native driver call you also get 0 documents:
const native = await db.collection("contents").find({}).toArray();

Note some items have been renamed.

Do you know why it happens?

no

-------- added 2026-02-14 8:42 AM EST --------------
If I aggregate the same pipeline above to the initial collection I can get the results queried, but not if I query the view. Also works on our staging db etc.

[hasezoey]

You are using MMS 11.x, and dont define a binary version, so i assume the default for that version 8.2.1.

---

This does not look to be actually MMS related and I have not used unionWith before so i cannot really help.
Please note that mongodb-memory-server (MMS for short), is just a mongodb server manager; it manages download, start & stop of the binaries (it does not re-implement mongodb).

Due to that, i will move this to be a discussion. (though you are likely better off asking on stackoverflow)

[franky-continu]

You are using MMS 11.x, and dont define a binary version, so i assume the default for that version 8.2.1.

I have not used unionWith before so i cannot really help. Please note that mongodb-memory-server (MMS for short), is just a mongodb server manager; it manages download, start & stop of the binaries ( it does not re-implement mongodb).

Due to that, i will move this to be a discussion. (though you are likely better off asking on stackoverflow)

Sure thing, thank you. And yes, version is 8.0.2 (I removed the code that printed the version)

const status = await mongoose.connection.db.admin().serverStatus();
console.log(status.version);

[franky-continu]

We found a kind of answer. Mongoose Model/Schema automatically creates a collection so we drop that and also tried adding autoCreate: false and autoIndex: false for the model definition.

{ autoCreate: false, autoIndex: false, autoSearchIndex: false, collection: 'contents' }

[franky-continu]

{ autoCreate: false, autoIndex: false, autoSearchIndex: false, collection: 'contents' }

Also double check with a drop statement if found.

const existing = await db.listCollections({ name: 'contents' }, { nameOnly: true }).toArray();
  if (existing.length > 0) {
    try {
      log.info(`View contents was found. Attempting to drop it and create it with the latest pipeline.`);
      const dropCollectionResult = await mongoose.connection.dropCollection(ContentsReadViewName);
      log.info(`View contents dropped with result`, dropCollectionResult);
    } catch (error) {
      // Handle other potential errors
      log.error(error, `Error bootstrapping: dropping view/collection: ${ error.message }`);
      throw error;
    }
  }

[hasezoey]

Good to see you found a solution, just for future reference, mongoose has a little documentation regarding view: Models: Views.

[franky-continu]

Yeah the issue for us was that we don't have a single collection like user in the Mongoose case sample, we have distributed contents and an existing view, but is definitely worth trying though perhaps unifying all distributed content in a single definition file

[franky-continu]

Thank you so so much for the info on createCollection, it actually did not work with the model but it did work with connection.

mongoose.connection.createCollection(...);

I thought you would need the reference of other models which you kind of could need, but it doesn't matter, seems to work,

I changed it really quick to test it, and all tests passed.

What we were doing or are doing, because I'm not going to use the createCollection, is using commands which allows me to drop it if I need to replace it and things of that nature, which for what I read in the doc it doesn't really update it.

In a very quick glance I dismissed it when I should have pursued it. In the end command {CREATE} and createCollection are quite similar.

the options:

{ autoCreate: false, autoIndex: false, autoSearchIndex: false, collection: 'contents' }

Are key not to have a collection created before you do it with command or createCollection.

Ich hätte Ihre Antwort genauer lesen sollen. Vielen Dank. :-)