(Orleans 9.1.2) Stateless woker grain potential thread pool tasks leak?

[vrecluse]

I'm using Orleans 9.1.2 in production environment, monitoring metrics counter dotnet.thread_pool.work_item.count.
After time elapsed, the counter grows faster and faster per minute.
This phenomenon disapeared after setting StatelessWorkerOptions.RemoveIdleWorkers to false, or remove all stateless workers.
Maybe it's related to the following code which been added after Orleans 9?

        if (strategy.RemoveIdleWorkers && options.RemoveIdleWorkers)
        {
            _minIdleCyclesBeforeRemoval = options.MinIdleCyclesBeforeRemoval > 0 ? options.MinIdleCyclesBeforeRemoval : 1;
            _inspectionTimer = new Timer(
                static state => ((StatelessWorkerGrainContext)state!).EnqueueWorkItem(WorkItemType.CollectIdleWorkers, CollectIdleWorkersSentinel),
                this,
                options.IdleWorkersInspectionPeriod,
                options.IdleWorkersInspectionPeriod);
        }

[gfix]

Perhaps the _inspectionTimer isn't disposed of properly? We (probably) see the same behavior where the number of dotnet_threadpool_timer_count increases until the silos starts timing out. The following graph shows the counter. An Orleans upgrade from version 9.1.2 to 9.2.1 was deployed at around 12:40, restarted at around 15:10, then reverted to 9.1.2 at around 15:30.

@vrecluse did you perhaps mean version 9.2.1, since the StatelessWorkerGrain change appeared in 9.2.0?

Any thoughts, @ledjon-behluli? 🙂

[vrecluse]

@vrecluse did you perhaps mean version 9.2.1, since the StatelessWorkerGrain change appeared in 9.2.0?

No, I'm using Orleans 9.1.2.

[ledjon-behluli]

Thanks @gfix , PR: #9636