Some thing wrong with add_histogram function

[ngocgiang99]

I had a strange problem when I tried to implement EfficientNet model. Last week, this code worked fine. But now, when I retrain then occurs some error. I faced this error in the validation step.

This is my code: https://github.com/ngocgiang99/Paper-Implementation. Please checkout to branch feat_efficient_net.

This is error log:

Traceback (most recent call last):
  File "train.py", line 73, in <module>
    main(config)
  File "train.py", line 54, in main
    trainer.train()
  File "D:\Work\Me\Paper-Implementation\efficient_net\base\base_trainer.py", l
    result = self._train_epoch(epoch)
  File "D:\Work\Me\Paper-Implementation\efficient_net\trainer\trainer.py", lin
    val_log = self._valid_epoch(epoch)
  File "D:\Work\Me\Paper-Implementation\efficient_net\trainer\trainer.py", lin
    self.writer.add_histogram(name, p, bins='auto')
  File "D:\Work\Me\Paper-Implementation\efficient_net\logger\visualization.py"
    add_data(tag, data, self.step, *args, **kwargs)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\torch\utils\tenso
in add_histogram
    histogram(tag, values, bins, max_bins=max_bins), global_step, walltime)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\torch\utils\tenso in histogram
    hist = make_histogram(values.astype(float), bins, max_bins)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\torch\utils\tenso in make_histogram
    counts, limits = np.histogram(values, bins=bins)
  File "<__array_function__ internals>", line 6, in histogram
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\numpy\lib\histogram
    bin_edges, uniform_bins = _get_bin_edges(a, bins, range, weights)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\numpy\lib\histogrn_edges
    endpoint=True, dtype=bin_type)
  File "<__array_function__ internals>", line 6, in linspace
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\numpy\core\function_base.py", line 135, in linspace
    y = _nx.arange(0, num, dtype=dt).reshape((-1,) + (1,) * ndim(delta))
numpy.core._exceptions.MemoryError: Unable to allocate 10.3 TiB for an array with shape (1418558411252,) and data type float64

Conda environment:

# Name                    Version                   Build  Channel
absl-py                   0.12.0                   pypi_0    pypi
ca-certificates           2021.4.13            haa95532_1
cachetools                4.2.2                    pypi_0    pypi
certifi                   2020.12.5        py37haa95532_0
chardet                   4.0.0                    pypi_0    pypi
google-auth               1.30.1                   pypi_0    pypi
google-auth-oauthlib      0.4.4                    pypi_0    pypi
grpcio                    1.38.0                   pypi_0    pypi
idna                      2.10                     pypi_0    pypi
importlib-metadata        4.3.1                    pypi_0    pypi
markdown                  3.3.4                    pypi_0    pypi
numpy                     1.20.3                   pypi_0    pypi
oauthlib                  3.1.0                    pypi_0    pypi
openssl                   1.1.1k               h2bbff1b_0
pandas                    1.2.4                    pypi_0    pypi
pillow                    8.2.0                    pypi_0    pypi
pip                       21.1.1           py37haa95532_0
protobuf                  3.17.1                   pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
python                    3.7.10               h6244533_0
python-dateutil           2.8.1                    pypi_0    pypi
pytz                      2021.1                   pypi_0    pypi
requests                  2.25.1                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rsa                       4.7.2                    pypi_0    pypi
setuptools                52.0.0           py37haa95532_0
six                       1.16.0                   pypi_0    pypi
sqlite                    3.35.4               h2bbff1b_0
tensorboard               2.5.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
torch                     1.8.1+cu111              pypi_0    pypi
torchaudio                0.8.1                    pypi_0    pypi
torchvision               0.9.1+cu111              pypi_0    pypi
tqdm                      4.61.0                   pypi_0    pypi
typing-extensions         3.10.0.0                 pypi_0    pypi
urllib3                   1.26.5                   pypi_0    pypi
vc                        14.2                 h21ff451_1
vs2015_runtime            14.27.29016          h5e58377_2
werkzeug                  2.0.1                    pypi_0    pypi
wheel                     0.36.2             pyhd3eb1b0_0
wincertstore              0.2                      py37_0
zipp                      3.4.1                    pypi_0    

Thanks.

[MohamedA95]

Hi,
I am not an expert but just out of curiosity is normal for it to try to allocate 10.3 TiB???

[ngocgiang99]

Hi @MohamedA95 , I printed the shape of p variable. The shape is small but I don't know why it allocate very large shape.
Here is log when I printed shape of p:

histogram shape:  torch.Size([32, 3, 3, 3])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32, 32, 1, 1])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32, 1, 3, 3])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([32])
histogram shape:  torch.Size([16, 32, 1, 1])
histogram shape:  torch.Size([16])
histogram shape:  torch.Size([16])
histogram shape:  torch.Size([16])
histogram shape:  torch.Size([96, 16, 1, 1])
histogram shape:  torch.Size([96])
histogram shape:  torch.Size([96])
histogram shape:  torch.Size([96])
histogram shape:  torch.Size([96, 1, 3, 3])
histogram shape:  torch.Size([96])
histogram shape:  torch.Size([96])
histogram shape:  torch.Size([96])
histogram shape:  torch.Size([24, 96, 1, 1])
histogram shape:  torch.Size([24])
histogram shape:  torch.Size([24])
histogram shape:  torch.Size([24])
histogram shape:  torch.Size([144, 24, 1, 1])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144, 1, 3, 3])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([24, 144, 1, 1])
histogram shape:  torch.Size([24])
histogram shape:  torch.Size([24])
histogram shape:  torch.Size([24])
histogram shape:  torch.Size([144, 24, 1, 1])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144, 1, 5, 5])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([144])
histogram shape:  torch.Size([40, 144, 1, 1])
histogram shape:  torch.Size([40])
histogram shape:  torch.Size([40])
histogram shape:  torch.Size([40])
histogram shape:  torch.Size([240, 40, 1, 1])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240, 1, 5, 5])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([40, 240, 1, 1])
histogram shape:  torch.Size([40])
histogram shape:  torch.Size([40])
histogram shape:  torch.Size([40])
histogram shape:  torch.Size([240, 40, 1, 1])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240, 1, 3, 3])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([240])
histogram shape:  torch.Size([80, 240, 1, 1])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([480, 80, 1, 1])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480, 1, 3, 3])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([80, 480, 1, 1])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([480, 80, 1, 1])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480, 1, 3, 3])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([80, 480, 1, 1])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([80])
histogram shape:  torch.Size([480, 80, 1, 1])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480, 1, 5, 5])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([480])
histogram shape:  torch.Size([112, 480, 1, 1])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([672, 112, 1, 1])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672, 1, 5, 5])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([112, 672, 1, 1])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([672, 112, 1, 1])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672, 1, 5, 5])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([112, 672, 1, 1])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([112])
histogram shape:  torch.Size([672, 112, 1, 1])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672, 1, 5, 5])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([672])
histogram shape:  torch.Size([192, 672, 1, 1])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([1152, 192, 1, 1])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152, 1, 5, 5])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([192, 1152, 1, 1])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([1152, 192, 1, 1])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152, 1, 5, 5])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([192, 1152, 1, 1])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([1152, 192, 1, 1])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152, 1, 5, 5])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([1152])
histogram shape:  torch.Size([192, 1152, 1, 1])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([192])
histogram shape:  torch.Size([1152, 192, 1, 1])
histogram shape:  torch.Size([1152])
Traceback (most recent call last):
  File "train.py", line 73, in <module>
    main(config)
  File "train.py", line 54, in main
    trainer.train()
  File "D:\Work\Me\Paper-Implementation\efficient_net\base\base_trainer.py", line 63, in train
    result = self._train_epoch(epoch)
  File "D:\Work\Me\Paper-Implementation\efficient_net\trainer\trainer.py", line 68, in _train_epoch
    val_log = self._valid_epoch(epoch)
  File "D:\Work\Me\Paper-Implementation\efficient_net\trainer\trainer.py", line 100, in _valid_epoch    self.writer.add_histogram(name, p, bins='auto')
  File "D:\Work\Me\Paper-Implementation\efficient_net\logger\visualization.py", line 65, in wrapper
    add_data(tag, data, self.step, *args, **kwargs)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\torch\utils\tensorboard\writer.py", line 429, in add_histogram
    histogram(tag, values, bins, max_bins=max_bins), global_step, walltime)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\torch\utils\tensorboard\summary.py", line 300, in histogram
    hist = make_histogram(values.astype(float), bins, max_bins)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\torch\utils\tensorboard\summary.py", line 309, in make_histogram
    counts, limits = np.histogram(values, bins=bins)
  File "<__array_function__ internals>", line 6, in histogram
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\numpy\lib\histograms.py", line 792, in 
histogram
    bin_edges, uniform_bins = _get_bin_edges(a, bins, range, weights)
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\numpy\lib\histograms.py", line 448, in 
_get_bin_edges
    endpoint=True, dtype=bin_type)
  File "<__array_function__ internals>", line 6, in linspace
  File "C:\Users\PC\anaconda3\envs\general\lib\site-packages\numpy\core\function_base.py", line 135, in linspace
    y = _nx.arange(0, num, dtype=dt).reshape((-1,) + (1,) * ndim(delta))
numpy.core._exceptions.MemoryError: Unable to allocate 10.3 TiB for an array with shape (1418558411252,) and data type float64 

[Infin1te2021]

Traceback (most recent call last):
  File "G:\DNN\train.py", line 89, in <module>
    main(config)
  File "G:\DNN\train.py", line 70, in main
    trainer.train()
  File "G:\DNN\base\base_trainer.py", line 63, in train
    result = self._train_epoch(epoch)
  File "G:\DNN\trainer\trainer.py", line 73, in _train_epoch
    val_log = self._valid_epoch(epoch)
  File "G:\DNN\trainer\trainer.py", line 117, in _valid_epoch
    self.writer.add_histogram(name, p, bins='auto')
  File "G:\DNN\logger\visualization.py", line 69, in wrapper
    add_data(tag, data, self.step, *args, **kwargs)
  File "C:\Users\zh398\AppData\Local\Programs\Python\Python310\lib\site-packages\tensorboardX\writer.py", line 562, in add_histogram
    histogram(tag, values, bins, max_bins=max_bins), global_step, walltime)
  File "C:\Users\zh398\AppData\Local\Programs\Python\Python310\lib\site-packages\tensorboardX\summary.py", line 211, in histogram
    return Summary(value=[Summary.Value(tag=name, histo=hist)])
MemoryError

Similar to me, sometimes I'm facing same error message. I was using the following module version

numpy==1.26.3
torch ==2.5.1+cu124
tensorboard==2.18.0

My memory started to explode during the 7th epoch validation phase, and my entire 64GB memory was occupied. Then I upgrade the numpy version to 2.2.2 and modified the code from

for name, p in self.model.named_parameters():
    self.writer.add_histogram(name, p, bins='auto')

to (Tensorboard writer add_histogram bug)

for name, p in self.model.named_parameters():
    try:
      self.writer.add_histogram(name, p, bins='auto')
    except:
      self.logger.warning(f"Failed to log histogram of {name} with min={p.min()} and max={p.max()} is {p}")

Still found the error message like

Failed to log histogram of transformer.layers.5.0.to_qkv.weight with min=-0.044450633227825165 and max=0.042818307876586914 is Parameter containing:
tensor([[ 6.9063e-11,  4.8692e-11,  3.9253e-09,  ...,  9.2091e-11,
         -6.8855e-09,  3.6025e-12],
        [ 1.5703e-11, -4.2516e-11, -6.1613e-11,  ..., -4.1147e-11,
          1.3427e-11, -5.6271e-11],
        [-1.2674e-10,  7.2862e-11,  9.1040e-11,  ..., -5.2313e-10,
         -9.1188e-11,  1.4888e-10],
        ...,
        [-1.0276e-05,  5.0609e-03, -1.9275e-02,  ...,  6.0142e-03,
         -8.3729e-03, -1.3289e-02],
        [-1.7562e-02, -1.6495e-02, -2.1187e-02,  ..., -1.8437e-02,
         -1.1596e-02, -1.5132e-02],
        [ 5.3506e-03, -1.1819e-02,  3.1767e-03,  ..., -2.0364e-02,
          1.4527e-02, -2.8218e-03]], device='cuda:0', requires_grad=True)

It all started when I changed the optimizer from AdamW to Adam. It runs correctly once I rolled it back.