unexpected keyword argument 'serialized_options' in some envs

[yaroslavvb]

Sometimes this is fixed by

000003e-05', '--n_layer', '18', '--d_model', '1024', '--n_head', '16', '--d_head', '64', '--d_inner', '4096', '--dropout', '0.1', '--dropatt', '0.1', '--optim', 'lamb', '--warmup_tokens', '0', '--tgt_len', '384', '--mem_len', '384', '--eval_tgt_len', '128', '--fp16', '--dynamic_loss_scale', '--init_std', '0.005', '--div_val', '1', '--bpe', '--checkpoint_each_epoch', '1', '--checkpoint', '/ncluster/runs/txl.09/model-best.pt', '--scheduler', 'constant']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "train.py", line 22, in <module>
    from tensorboardX import SummaryWriter
  File "/home/ubuntu/anaconda3/envs/pytorch_april/lib/python3.6/site-packages/tensorboardX/__init__.py", line 5, in <module>
    from .torchvis import TorchVis
  File "/home/ubuntu/anaconda3/envs/pytorch_april/lib/python3.6/site-packages/tensorboardX/torchvis.py", line 11, in <module>
    from .writer import SummaryWriter
  File "/home/ubuntu/anaconda3/envs/pytorch_april/lib/python3.6/site-packages/tensorboardX/writer.py", line 27, in <module>
    from .event_file_writer import EventFileWriter
  File "/home/ubuntu/anaconda3/envs/pytorch_april/lib/python3.6/site-packages/tensorboardX/event_file_writer.py", line 28, in <module>
    from .proto import event_pb2
  File "/home/ubuntu/anaconda3/envs/pytorch_april/lib/python3.6/site-packages/tensorboardX/proto/event_pb2.py", line 15, in <module>
    from tensorboardX.proto import summary_pb2 as tensorboardX_dot_proto_dot_summary__pb2
  File "/home/ubuntu/anaconda3/envs/pytorch_april/lib/python3.6/site-packages/tensorboardX/proto/summary_pb2.py", line 15, in <module>
    from tensorboardX.proto import tensor_pb2 as tensorboardX_dot_proto_dot_tensor__pb2
  File "/home/ubuntu/anaconda3/envs/pytorch_april/lib/python3.6/site-packages/tensorboardX/proto/tensor_pb2.py", line 15, in <module>
    from tensorboardX.proto import resource_handle_pb2 as tensorboardX_dot_proto_dot_resource__handle__pb2
  File "/home/ubuntu/anaconda3/envs/pytorch_april/lib/python3.6/site-packages/tensorboardX/proto/resource_handle_pb2.py", line 22, in <module>
    serialized_pb=_b('\n(tensorboardX/proto/resource_handle.proto\x12\x0ctensorboardX\"r\n\x13ResourceHandleProto\x12\x0e\n\x06\x64\x65vice\x18\x01 \x01(\t\x12\x11\n\tcontainer\x18\x02 \x01(\t\x12\x0c\n\x04name\x18\x03 \x01(\t\x12\x11\n\thash_code\x18\x04 \x01(\x04\x12\x17\n\x0fmaybe_type_name\x18\x05 \x01(\tB/\n\x18org.tensorflow.frameworkB\x0eResourceHandleP\x01\xf8\x01\x01\x62\x06proto3')
TypeError: __new__() got an unexpected keyword argument 'serialized_options'
(pytorch_april) ubuntu@ip-172-31-44-63:~$ source activate pytorch_p36
(pytorch_p36) ubuntu@ip-172-31-44-63:~$ NCCL_DEBUG=VERSION NCCL_MIN_NRINGS=16 NCCL_MAX_NRINGS=16  python -m torch.distributed.launch --nproc_per_node=8 --nnodes=1 --node_rank=0 --master_addr=172.31.44.63 --master_port=6016 train.py --seed 1111 --data data/wikiextracted --dataset wiki  --log_interval 100 --eval_interval 500 --max_tokens 1500000000 --logdir /ncluster/runs/wiki.09 --lr 0.000375 --batch_size 6 --eta_min 3.7500000000000003e-05 --n_layer 18 --d_model 1024 --n_head 16 --d_head 64 --d_inner 4096 --dropout 0.1 --dropatt 0.1 --optim lamb --warmup_tokens 0 --tgt_len 384 --mem_len 384 --eval_tgt_len 128 --fp16 --dynamic_loss_scale --init_std 0.005 --div_val 1 --bpe --checkpoint_each_epoch 1 --checkpoint /ncluster/runs/txl.09/model-best.pt --scheduler constant > >(tee -a /tmp/ncluster/0.wiki-dpwp8/10.out) 2> >(tee -a /tmp/ncluster/0.wiki-dpwp8/10.out >&2); echo $? > /tmp/ncluster/0.wiki-dpwp8/

[yaroslavvb]

This is same issue as tensorflow/models#3995, requires uninstall/reinstall of protobuf

[yaroslavvb]

pip uninstall protobuf
pip install -U protobuf

[Usmaniatech]

@yaroslavvb