Add Optional Batched Yielding to `QuerySet.iterator()` and `QuerySet.aiterator()`

[JohananOppongAmoateng]

Code of Conduct

• I agree to follow Django's Code of Conduct

Feature Description

This proposal suggests extending Django’s QuerySet.iterator() and QuerySet.aiterator() methods with an optional parameter (e.g. batch_yield_size) that allows them to yield lists of model instances in batches, rather than yielding one object at a time.

The goal is to combine the memory efficiency of streaming querysets with the convenience of grouped (batched) processing.

Problem

Currently, QuerySet.iterator() and QuerySet.aiterator() always yield one model instance per iteration. While this is ideal for memory efficiency, it becomes inconvenient or inefficient for common batch-oriented workflows, such as:

• Bulk writes to external systems (APIs, message queues, file exports)
• Batch validation or transformation pipelines
• Chunked background processing
• Network or I/O bound operations where per-object overhead is significant

Developers often work around this by manually buffering results into lists, for example:

buffer = []
for obj in queryset.iterator():
    buffer.append(obj)
    if len(buffer) == 1000:
        process(buffer)
        buffer.clear()

This pattern is repetitive, error-prone, and obscures intent. Django already supports database-level chunking via chunk_size, but there is no built-in way to express logical batch processing at the iteration level.

Request or proposal

proposal

Additional Details

This feature is orthogonal to chunk_size, which controls how many rows are fetched from the database cursor.

It differs from Paginator in that:

It does not require counting the full queryset.

It supports true streaming over large datasets.

The feature is especially useful for async workflows, where per-object await overhead is high and batching significantly improves throughput.

The default behavior would remain unchanged; batching would be opt-in.

Implementation Suggestions

Synchronous Example

for batch in MyModel.objects.iterator(

    chunk_size=2000,
    batch_yield_size=500,
):
    send_to_api(batch)

Proposed behavior:

Internally fetch rows using chunk_size

Yield lists of up to batch_yield_size model instances

The final batch may be smaller

Asynchronous Example

async for batch in MyModel.objects.aiterator(
    chunk_size=2000,
    batch_yield_size=500,
):
    await send_to_api(batch)

[github-actions]

Thank you JohananOppongAmoateng for sharing your idea! We have a lot of them so please be patient. You can see the current queue here.

Community instructions

For commenters, please use the emoji reactions on the issue to express support, and/or concern easily. Please use the comments to ask questions or contribute knowledge about the idea. It is unhelpful to post comments of "I'd love this" or "What's the state of this?"

Reaction Guide

• 👍 This is something I would use
• 👎 This is something that would cause problems for me or Django
• 😕 I’m indifferent to this
• 🎉 This is an easy win

[RealOrangeOne]

Python 3.12 introduced itertools.batched which handles the batching side for you:

for buffer in itertools.batched(queryset.iterator(), 500):
    pass

What would adding a Django native API bring? (besides async support)

[andrew-cybsafe]

For Python < 3.12, more_itertools provides an implementation of batched (as well as other goodies).

[tim-schilling]

@RealOrangeOne Maybe we should add an example of this to the documentation to the "Models, data and databases" in the how to guides section?

Python 3.12 introduced itertools.batched which handles the batching side for you:

for buffer in itertools.batched(queryset.iterator(), 500):
    pass

[RealOrangeOne]

Documentation might be a good way to handle this. However, I'm not sure if it's a little heavy-handled for what is really just a 2-line snippet.

[tim-schilling]

I suspect the explanation of when you'd need to use this and why it's useful will be a couple of paragraphs. Yes, it's very little code and combines built-in tools, but it's a great technique for efficient data retrieval.