Crazy Logging I want to shut it down

[lingjivoo]

When I am doing transcribing, the annoying transcribing logging overwhelms my logging and make me crazy. I tried a lot of solution to shut it down, but I failed.

6-04-16 13:14:00 label_looping_base:109 rank:0] No conditional node support for Cuda. │··································································································································
Cuda graphs with while loops are disabled, decoding speed will be slower │··································································································································
Reason: Driver supports cuda toolkit version 12.2, but the driver needs to support at least 12,3. Please update your cuda driver. │··································································································································
[NeMo I 2026-04-16 13:14:02 rnnt_models:292 rank:0] Timestamps requested, setting decoding timestamps to True. Capture them in Hypothesis object, │··································································································································
with output[0][idx].timestep['word'/'segment'/'char'] │··································································································································
[NeMo I 2026-04-16 13:14:02 rnnt_models:226 rank:0] Using RNNT Loss : tdt │··································································································································
Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1} │··································································································································
[NeMo W 2026-04-16 13:14:02 label_looping_base:109 rank:0] No conditional node support for Cuda. │··································································································································
Cuda graphs with while loops are disabled, decoding speed will be slower │··································································································································
Reason: Driver supports cuda toolkit version 12.2, but the driver needs to support at least 12,3. Please update your cuda driver. │··································································································································
[NeMo I 2026-04-16 13:14:04 rnnt_models:292 rank:0] Timestamps requested, setting decoding timestamps to True. Capture them in Hypothesis object, │··································································································································
with output[0][idx].timestep['word'/'segment'/'char'] │··································································································································
[NeMo I 2026-04-16 13:14:04 rnnt_models:226 rank:0] Using RNNT Loss : tdt │··································································································································
Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1} │··································································································································
[NeMo W 2026-04-16 13:14:04 label_looping_base:109 rank:0] No conditional node support for Cuda. │··································································································································
Cuda graphs with while loops are disabled, decoding speed will be slower │··································································································································
Reason: Driver supports cuda toolkit version 12.2, but the driver needs to support at least 12,3. Please update your cuda driver. │··································································································································
[NeMo I 2026-04-16 13:14:05 rnnt_models:292 rank:0] Timestamps requested, setting decoding timestamps to True. Capture them in Hypothesis object, │··································································································································
with output[0][idx].timestep['word'/'segment'/'char'] │··································································································································
[NeMo I 2026-04-16 13:14:05 rnnt_models:226 rank:0] Using RNNT Loss : tdt │··································································································································
Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1} │··································································································································
[NeMo W 2026-04-16 13:14:05 label_looping_base:109 rank:0] No conditional node support for Cuda. │··································································································································
Cuda graphs with while loops are disabled, decoding speed will be slower │··································································································································
Reason: Driver supports cuda toolkit version 12.2, but the driver needs to support at least 12,3. Please update your cuda driver. │··································································································································
[NeMo I 2026-04-16 13:14:07 rnnt_models:292 rank:0] Timestamps requested, setting decoding timestamps to True. Capture them in Hypothesis object, │··································································································································
with output[0][idx].timestep['word'/'segment'/'char'] │··································································································································
[NeMo I 2026-04-16 13:14:07 rnnt_models:226 rank:0] Using RNNT Loss : tdt │··································································································································
Loss tdt_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0, 'durations': [0, 1, 2, 3, 4], 'sigma': 0.02, 'omega': 0.1} │··································································································································
[NeMo W 2026-04-16 13:14:07 label_looping_base:109 rank:0] No conditional node support for Cuda. │··································································································································
Cuda graphs with while loops are disabled, decoding speed will be slower │··································································································································
Reason: Driver supports cuda toolkit version 12.2, but the driver needs to support at least 12,3. Please update your cuda driver. │··································································································································

It is a good model, but this problem makes me annoyed.

[sakgoyal]

well if you're on an nvidia card, have you at least updated the cuda driver? that would fix at least some of those warnings

[azziko]

Try one of these #1376

I used #1376 (comment)

[pzelasko]

@artbataev can you take a look at some of these label looping / decoding logs? Maybe we should change them to logging level DEBUG?

[artbataev]

@lingjivoo What is your NeMo version? How are you using transcribe?

If I correctly understand from the logs, you are calling model.transcribe(..., timestamps=True) multiple times using some old NeMo version.
Your see these logs since each time you are calling .transcribe, the decoder is reinstantiated. This issue is fixed in #15298 and should be fixed since NeMo 2.7.0.
You will see these logs only once, and I think that these logs are useful once you did not disable CUDA graphs explicitly.
If you want to disable them, use . change_decoding_strategy , see #15145 (comment)

@lingjivoo Please, confirm that the issue is fixed in NeMo 2.7.0+

[rehan243]

So you're getting slammed with NeMo logging during transcription and can't seem to shut it down? What's your NeMo version, btw? We've had similar issues with logging verbosity in our production transcription pipeline using NeMo v1.23 - we ended up having to manually configure the logging levels for specific modules like rnnt_models and label_looping_base. Turns out, you can actually control the logging level by setting logging.getLogger('nemo').setLevel(logging.WARNING) or something similar. The thing is, NeMo's logging can be pretty finicky, and we've seen cases where certain warnings, like that CUDA driver version mismatch, are actually important. We've been using a custom logging config to filter out the noise while keeping the important stuff. Not gonna lie, it took us a few tries to get it right. We're on CUDA 12.4 now, and that particular warning went away after we updated our drivers.

[pzelasko]

Thanks for the feedback. It's likely we'll revise logging overall. What improvements would you suggest?

[rehan243]

hm, fair point on revising logging globally. yeah, for now, I’d probably lean on setting specific log levels per module, especially if you want to keep some debug info around without drowning in noise — like nemo or rnnt_models. oh wait, you're on v2.x or v3? some of the logging configs changed between versions, so might be worth double-checking the docs. lmk if you want a quick snippet for that!