I want to process multiple inference requests and evaluate the the average request latency. Because increasing the batch size will add extra latency to the request, I want to find the optimal maximun batch size to minimize the average average request latency.
So instead of letting vllm decide batch size at each iteration, is there a way to specify the max batch size (e.g., 40 requests inference at one iteration at most) with continous batching?
I want to process multiple inference requests and evaluate the the average request latency. Because increasing the batch size will add extra latency to the request, I want to find the optimal maximun batch size to minimize the average average request latency.
So instead of letting vllm decide batch size at each iteration, is there a way to specify the max batch size (e.g., 40 requests inference at one iteration at most) with continous batching?