THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/THCTensorRandom.cu line=25 error=30 : unknown error

[NJUSTghw]

loading all datasets...
using 4 threads
loading from cache file: ./cache/coco_trainval2014.pkl
loading annotations into memory...
Done (t=11.78s)
creating index...
index created!
loading from cache file: ./cache/coco_trainval2014.pkl
loading annotations into memory...
Done (t=8.33s)
creating index...
index created!
loading from cache file: ./cache/coco_trainval2014.pkl
loading annotations into memory...
Done (t=7.80s)
creating index...
index created!
loading from cache file: ./cache/coco_trainval2014.pkl
loading annotations into memory...
Done (t=11.06s)
creating index...
index created!
loading from cache file: ./cache/coco_minival2014.pkl
loading annotations into memory...
Done (t=0.38s)
creating index...
index created!
system config...
{'batch_size': 2,
'cache_dir': './cache',
'chunk_sizes': [4, 5, 5, 5, 5, 5, 5, 5, 5, 5],
'config_dir': './config',
'data_dir': './data',
'data_rng': <mtrand.RandomState object at 0x7f6494e26a68>,
'dataset': 'MSCOCO',
'decay_rate': 10,
'display': 5,
'learning_rate': 0.00025,
'max_iter': 500000,
'nnet_rng': <mtrand.RandomState object at 0x7f6494e26ab0>,
'opt_algo': 'adam',
'prefetch_size': 5,
'pretrain': None,
'result_dir': './results',
'sampling_function': 'kp_detection',
'snapshot': 5000,
'snapshot_name': 'CornerNet',
'stepsize': 450000,
'test_split': 'testdev',
'train_split': 'trainval',
'val_iter': 100,
'val_split': 'minival',
'weight_decay': False,
'weight_decay_rate': 1e-05,
'weight_decay_type': 'l2'}
db config...
{'ae_threshold': 0.5,
'border': 128,
'categories': 80,
'data_aug': True,
'gaussian_bump': True,
'gaussian_iou': 0.7,
'gaussian_radius': -1,
'input_size': [511, 511],
'lighting': True,
'max_per_image': 100,
'merge_bbox': False,
'nms_algorithm': 'exp_soft_nms',
'nms_kernel': 3,
'nms_threshold': 0.5,
'output_sizes': [[128, 128]],
'rand_color': True,
'rand_crop': True,
'rand_pushes': False,
'rand_samples': False,
'rand_scale_max': 1.4,
'rand_scale_min': 0.6,
'rand_scale_step': 0.1,
'rand_scales': array([0.6, 0.7, 0.8, 0.9, 1. , 1.1, 1.2, 1.3]),
'special_crop': False,
'test_scales': [1],
'top_k': 100,
'weight_exp': 8}
len of db: 118287
start prefetching data...
shuffling indices...
start prefetching data...
shuffling indices...
start prefetching data...
shuffling indices...
start prefetching data...
shuffling indices...
(480, 640, 3)
(480, 640, 3)
(427, 640, 3)
(434, 640, 3)
building model...
module_file: models.CornerNet
start prefetching data...
shuffling indices...
(480, 640, 3)
(375, 500, 3)
(480, 640, 3)
(427, 640, 3)
(333, 500, 3)
(640, 523, 3)
(424, 640, 3)
(480, 640, 3)
(375, 500, 3)
(500, 334, 3)
(480, 640, 3)
(500, 375, 3)
(416, 640, 3)
(500, 375, 3)
(425, 640, 3)
(480, 640, 3)
(375, 500, 3)
(640, 427, 3)
(640, 609, 3)
(383, 640, 3)
(428, 640, 3)
(426, 640, 3)
(427, 640, 3)
(640, 427, 3)
(640, 480, 3)
(555, 640, 3)
(480, 640, 3)
(416, 640, 3)
(375, 500, 3)
(375, 500, 3)
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/THCTensorRandom.cu line=25 error=30 : unknown error

[NJUSTghw]

has anyone met the same issue? please help me out?

[NJUSTghw]

loading all datasets...
using 4 threads
loading from cache file: ./cache/coco_trainval2014.pkl
loading annotations into memory...
Done (t=8.17s)
creating index...
index created!
loading from cache file: ./cache/coco_trainval2014.pkl
loading annotations into memory...
Done (t=7.94s)
creating index...
index created!
loading from cache file: ./cache/coco_trainval2014.pkl
loading annotations into memory...
Done (t=7.52s)
creating index...
index created!
loading from cache file: ./cache/coco_trainval2014.pkl
loading annotations into memory...
Done (t=10.59s)
creating index...
index created!
loading from cache file: ./cache/coco_minival2014.pkl
loading annotations into memory...
Done (t=0.24s)
creating index...
index created!
system config...
{'batch_size': 1,
'cache_dir': './cache',
'chunk_sizes': [1],
'config_dir': './config',
'data_dir': './data',
'data_rng': <mtrand.RandomState object at 0x7f562d9ec990>,
'dataset': 'MSCOCO',
'decay_rate': 10,
'display': 5,
'learning_rate': 0.00025,
'max_iter': 500000,
'nnet_rng': <mtrand.RandomState object at 0x7f562d9ec9d8>,
'opt_algo': 'adam',
'prefetch_size': 5,
'pretrain': None,
'result_dir': './results',
'sampling_function': 'kp_detection',
'snapshot': 5000,
'snapshot_name': 'CornerNet',
'stepsize': 450000,
'test_split': 'testdev',
'train_split': 'trainval',
'val_iter': 100,
'val_split': 'minival',
'weight_decay': False,
'weight_decay_rate': 1e-05,
'weight_decay_type': 'l2'}
db config...
{'ae_threshold': 0.5,
'border': 128,
'categories': 80,
'data_aug': True,
'gaussian_bump': True,
'gaussian_iou': 0.7,
'gaussian_radius': -1,
'input_size': [511, 511],
'lighting': True,
'max_per_image': 100,
'merge_bbox': False,
'nms_algorithm': 'exp_soft_nms',
'nms_kernel': 3,
'nms_threshold': 0.5,
'output_sizes': [[128, 128]],
'rand_color': True,
'rand_crop': True,
'rand_pushes': False,
'rand_samples': False,
'rand_scale_max': 1.4,
'rand_scale_min': 0.6,
'rand_scale_step': 0.1,
'rand_scales': array([0.6, 0.7, 0.8, 0.9, 1. , 1.1, 1.2, 1.3]),
'special_crop': False,
'test_scales': [1],
'top_k': 100,
'weight_exp': 8}
len of db: 118287
start prefetching data...
shuffling indices...
start prefetching data...
shuffling indices...
start prefetching data...
shuffling indices...
start prefetching data...
shuffling indices...
start prefetching data...
building model...
module_file: models.CornerNet
shuffling indices...
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/THCTensorRandom.cu line=25 error=30 : unknown error
Exception in thread Thread-1:
Traceback (most recent call last):
File "/home/s106/anaconda3/envs/CornerNet/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/s106/anaconda3/envs/CornerNet/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "train.py", line 52, in pin_memory
data["xs"] = [x.pin_memory() for x in data["xs"]]
File "train.py", line 52, in
data["xs"] = [x.pin_memory() for x in data["xs"]]
RuntimeError: cuda runtime error (30) : unknown error at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/THCTensorRandom.cu:25

I am not sure what the exact error is

[Zhangyongtao123]

I meet this error, but I don't know how to solve it

[Zhangyongtao123]

Maybe it's just because you don't have enough GPUs or your GPUs don't have enough memory??!,I guess