Low accuracy of own trained model

[yichunk]

Hi ronghang

I am trying to train the high resolution model using your code. I followed all the instructions in the README, and did not change any parameters in the code, but the the performance of trained model is extremely low, just about 4.5% for overall IoU. Is it possible that you updated the code afterward, but the modified code is not uploaded to the github?

Thanks

[ronghanghu]

This is unexpected, and is probably due to the change in tensorflow versions (the code is written with TF 0.7.0 and is probably not compatible with newer version of tensorflow). If you download the pre-trained models and test their accuracy, do you get reasonable performances?

[yichunk]

Thanks for response. My TensorFlow is version 0.12. The pre-trained models achieved similar result as the paper. Actually, the accuracy is reasonable at the beginning of high_res training initialized with low_res, but it goes down when the number of iterations increases.

[ronghanghu]

Mmm, that's very strange. Probably the training batches weren't correctly built?

Blow is my terminal output when re-training the model using this repo. The output when training the high resolution model is in this gist:
https://gist.githubusercontent.com/ronghanghu/a98c3e28ef6a5c5b87c38e64034147eb/raw/2bddf0ecb3a9fb67b819e2859982b6cdd81648d9/exp_train_referit_seg_highres_log.txt
Does it look like your terminal output?

[yichunk]

well, the output is different a lot from each other. train_referit_seg_highres.log

[ronghanghu]

I'm not exactly sure what's wrong. The final accuracy 0.238127 (all), 0.418813 (pos), 0.216635 (neg) you got seems even lower than random guess (0.50 on this binary classification problem) and is quite weird. So I suspect there could be some data issue (training batches isn't correctly created). Perhaps re-building the training batches (via python exp-referit/build_training_batches_seg.py) can solve the issue.

[ronghanghu]

Btw, if you are in a hurry and the problem persists, you can try out the Caffe implementation of this project (https://github.com/Seth-Park/text_objseg_caffe implemented by Seth Park), which gives higher performance with a slightly different set of hyperparameters.

[yichunk]

Thanks for your kind help. I'll try the caffe implementation later.

[yichunk]

Seems the CUDA/GPU issue tensorflow/tensorflow#3507, though after upgrading CUDA version to 8.0, the issue is not resolved yet. I am trying to train on CPU only.